RAPO-RL-TAC: Risk-Aware Partial-Order Reinforcement Learning with Timed Automata Completion for the Interval Job Shop Problem

Abstract

In the Interval Job Shop Problem (IJSP), operation processing times are represented by intervals, making machine-sequencing decisions sensitive to temporal uncertainty. We propose Risk-Aware Partial-Order Reinforcement Learning with Timed Automata Completion (RAPO-RL-TAC), which couples Risk-Aware Partial-Order Reinforcement Learning (RAPO-RL) for machine-order construction with Timed Automata Completion (TAC) for execution-time resolution of deferred sequencing decisions. Risk awareness focuses on preserving temporal flexibility in machine-order relations whose preferred ordering is sensitive to interval uncertainty. RAPO-RL selectively commits comparatively determinate machine conflicts while retaining a bounded set of timing-sensitive relations. TAC completes unresolved relations as execution evolves, while statistical model checking characterises completion-time variability across timed executions. On 14 ORB and LA benchmark instances, RAPO-RL-TAC achieves mean midpoint makespans 2.92% and 1.63% lower than population-based neighbourhood search (PNS) and genetic algorithm (GA), respectively. Compared with reproduced Fast Elitist Artificial Bee Colony (fEABC) variants, RAPO-RL-TAC achieves a lower midpoint than at least one variant on 7 of 14 instances. In the controlled ablation study, RAPO-RL-TAC achieves a 13.85% lower mean midpoint makespan than hard enforcement of the learned relations. These results indicate that risk-aware selective commitment preserves temporal flexibility while maintaining competitive nominal schedule quality under interval uncertainty.

1. Introduction

The Job Shop Scheduling Problem (JSSP) concerns the assignment and sequencing of operations on shared machines while satisfying technological precedence and machine-capacity constraints [1,2,3]. Minimising the makespan is one of the most widely studied objectives in JSSP optimisation. Due to its combinatorial nature, the deterministic JSSP is NP-hard, and the construction of effective machine-order representations remains a fundamental issue in both exact and approximate scheduling methods. In practice, processing times are rarely known exactly, and reliable probability distributions are often unavailable, but lower and upper duration bounds can usually be estimated effectively. The Interval Job Shop Problem (IJSP) therefore models processing-time uncertainty by representing each operation duration as an interval defined by its lower and upper bounds [4]. Unlike formulations that rely on explicit probability distributions, such as chance-constrained scheduling [5], the IJSP can therefore represent processing-time uncertainty using only these interval bounds.

Most IJSP methods adopt a sequence-first strategy in which a complete operation or machine sequence is determined before interval durations are propagated for evaluation [6,7,8]. Robust scheduling research, by contrast, indicates that retaining selected sequencing freedom can support adaptability under uncertain durations [9,10,11]. This contrast motivates a representation that fixes sufficiently stable machine conflicts while deferring timing-sensitive relations to execution.

The key challenge is therefore to determine which machine conflicts should be committed before execution and which should remain open. Resolving all same-machine relations removes potentially useful temporal flexibility, whereas retaining too many leaves an unnecessarily large residual decision space. The resulting problem is to balance combinatorial reduction against flexibility under interval uncertainty.

To address this problem, we propose Risk-Aware Partial-Order Reinforcement Learning with Timed Automata Completion (RAPO-RL-TAC). Its learning component, Risk-Aware Partial-Order Reinforcement Learning (RAPO-RL), constructs a machine partial order by resolving comparatively stable conflicts while retaining a bounded number of timing-sensitive relations. Timed Automata Completion (TAC) subsequently resolves the deferred relations as execution evolves, after which statistical model checking characterises completion-time distributions and temporal variability.

The main contributions are as follows:

1.

Reinforcement learning for partial-order construction in the IJSP. Rather than learning a complete machine sequence, RAPO-RL learns which machine-conflict relations should be committed before execution, prioritising conflicts that are comparatively less sensitive to interval uncertainty while retaining temporally ambiguous relations for later resolution. The resulting directed partial order eliminates a substantial portion of the combinatorial ambiguity before formal temporal analysis, thereby reducing the residual branching space and improving the tractability of model-checking-based scheduling for the IJSP.

2.

A risk-aware selective commitment algorithm under interval uncertainty. We introduce a risk-aware selective-commitment mechanism that explicitly accounts for interval-induced ordering uncertainty when determining which machine conflicts should be fixed before execution. A temporal-determinacy criterion characterises each unresolved conflict using interval-derived execution-window overlap and ordering criticality, while a machine-wise open-edge budget limits the number of deferred relations. By committing comparatively stable conflicts and retaining timing-sensitive conflicts whose ordering is more susceptible to processing-time realisations, the risk-aware mechanism avoids premature commitment and preserves temporal flexibility while controlling the residual decision space.

3.

A timed-automata completion framework for the IJSP. We formulate residual machine-order uncertainty and interval processing times within a timed-automata model in which unresolved sequencing relations are completed according to the evolving temporal state. This formulation connects the learned partial order to executable schedules without requiring all machine orders to be fixed in advance; statistical model checking is then used to sample timed executions and characterise completion-time distributions and variability under interval uncertainty.

The remainder of the paper is organised as follows. Section 2 provides a comprehensive review of the related work. Section 3 defines the IJSP and presents the overall RAPO-RL-TAC framework. Section 4 details the reinforcement learning for partial-order construction and the risk-aware selective commitment algorithm, and Section 5 presents the method of timed automata completion and statistical temporal assessment. Section 6 reports the benchmark, ablation, and statistical-model-checking experiments. Section 7 concludes the paper.

2. Related Work

Existing studies relevant to solving job-shop problems under interval or other processing-time uncertainty can be organised into four methodological directions: metaheuristic optimisation, machine-learning-based scheduling, formal methods for JSSP scheduling, and hybrid methods. Metaheuristic optimisation constructs schedules under uncertain durations using neighbourhood, evolutionary, and swarm-intelligence approaches, with robustness-oriented criteria adopted where appropriate to evaluate uncertainty sensitivity; machine-learning-based scheduling learns scheduling or dispatching decisions from structured shop states; formal methods for JSSP scheduling use state-transition models and verification techniques to represent and analyse temporal scheduling behaviour; and hybrid methods combine data-driven decision making with explicit formal or event-based models. Representative foundations include interval and robustness-oriented optimisation methods [12,13], graph- and reinforcement-learning-based schedulers [14,15,16,17], robustness-evaluation methodology [18], and broader JSSP taxonomies [19]. This organisation classifies the literature by solution mechanism rather than using the IJSP itself as a research-direction label.

2.1. Metaheuristic Optimisation Under Processing-Time Uncertainty

Metaheuristic optimisation is a principal line of work for the IJSP because interval-valued processing times can be incorporated directly into schedule evaluation. Lei [6] developed population-based neighbourhood search (PNS) using an ordered operation-based representation and interval-makespan evaluation. Díaz et al. [7] subsequently proposed a genetic algorithm (GA) in which candidate sequences are decoded and evaluated under interval uncertainty. Artificial-bee-colony and neighbourhood-search variants have also been developed for interval job shops with flexible maintenance, dual-resource and environmental constraints, and makespan objectives [12,20,21,22]. Díaz et al. [8] later proposed the Fast Elitist Artificial Bee Colony (fEABC) algorithm, simplifying the underlying artificial bee colony structure to reduce runtime while retaining competitive interval-makespan performance. Afşar et al. [23] extended interval-based optimisation to a green flexible job-shop setting through a hybrid evolutionary approach with lexicographic production and environmental objectives.

Complementary uncertainty-aware studies evaluate robustness and risk rather than only a nominal interval makespan. Beck and Wilson [24] studied proactive job-shop scheduling with probabilistic activity durations. Wang et al. [25] formulated bad-scenario-set robust job-shop scheduling under processing-time uncertainty and solved it with tabu search. For interval-valued blocking job shops, Meloni et al. [26] evaluated makespan risk using value-at-risk and conditional value-at-risk. Calamita et al. [27] subsequently used machine learning to accelerate such risk estimation, while Himmiche et al. [18] presented a broader robustness-evaluation process for schedules under uncertainty.

A related robustness-oriented line preserves sequencing flexibility instead of fixing every resource order in advance. Earlier work studied graph-based decomposition, solution flexibility, and partial-order schedules as mechanisms for retaining alternatives under perturbations [9,10,28,29]. Rivière et al. [11] proposed a two-stage stochastic/robust method based on permutable operation groups, allowing part of the sequencing decision to be deferred until uncertainty is realised. Ourari et al. [30] more recently developed a robust job-shop scheduling approach based on sequential flexibility, retaining alternative machine sequences that can be adapted when disturbances occur during execution.

Taken together, these studies establish metaheuristic optimisation, explicit robustness/risk evaluation, and delayed sequencing as complementary responses to processing-time uncertainty. Most direct IJSP optimisers nevertheless return a complete sequence before temporal execution, whereas flexibility-oriented robust methods preserve alternatives without learning which individual machine conflicts should remain unresolved. This distinction motivates the selective partial-order representation used in RAPO-RL-TAC.

2.2. Machine-Learning-Based Scheduling

Machine-learning-based scheduling increasingly uses learned policies to construct job-shop decisions from structured shop states. Zhang et al. [14] introduced Learning to Dispatch, representing the evolving JSSP as a disjunctive graph and using a graph neural network (GNN) with Proximal Policy Optimisation (PPO) to learn priority dispatching rules. Park et al. [15] combined graph neural networks with reinforcement learning for joint representation and policy learning, and demonstrated transfer to previously unseen JSSP instances. Broader methodological perspectives on machine learning for combinatorial optimisation and the reliability of reinforcement-learning-based production scheduling are provided by Bengio et al. [31] and Waubert de Puiseau et al. [32], respectively. Cai et al. [33] and Yang et al. [34] further illustrate deep-reinforcement-learning and graph-based approaches in static and dynamic job-shop settings.

Uncertain processing times have also been incorporated directly into learned scheduling policies. Wu et al. [16] formulated dynamic job-shop scheduling with uncertain processing times as a reinforcement-learning problem using priority-rule actions and policy reuse under changing durations. Infantes et al. [17] addressed JSSP scheduling with uncertain activity durations and used deep reinforcement learning to search for robust schedules while studying generalisation and scalability. These studies show that uncertainty can enter the learning process itself rather than being considered only after a complete deterministic schedule has been generated.

Recent surveys reinforce this methodological trend. Lv et al. [35] review deep-reinforcement-learning approaches to job-shop scheduling, Smit et al. [36] focus on graph neural networks for JSSP, and Xu et al. [37] survey learning-to-optimise methods with comparisons between genetic programming and reinforcement learning. Meilanitasari and Shin [38] provide a complementary review of prediction and optimisation for sequence-driven scheduling in job-shop flexible manufacturing systems.

Despite differences in state representation and learning architecture, these methods generally use learning to select dispatching actions, operations, or progressively complete a schedule. Comparatively less attention has been paid to policies that deliberately return an incomplete machine-order representation in which only selected conflict relations are committed and the remaining relations are transferred to a separate temporal-completion stage.

2.3. Formal Methods for JSSP Scheduling

Formal methods provide mathematically defined state-transition models for expressing timing, synchronisation, and resource constraints. Timed automata extend finite-state models with real-valued clocks [39]; tool-supported timed-automata analysis and simulation-based statistical model checking (SMC) provide practical mechanisms for reachability analysis, temporal-property checking, and statistical estimation [40,41]. At this methodological level, the emphasis is on the formal modelling and analysis paradigm rather than on any particular software implementation.

Formal temporal models have been applied directly to scheduling and robustness analysis. Behrmann et al. [42] formulated production scheduling as reachability analysis over timed-automata models. Malik and Pena [43] used model checking to compute time-optimal task schedules in a flexible manufacturing system. Bendík et al. [44] analysed the robustness of timed automata under perturbations of timing constants using model-checking techniques. In production scheduling under uncertainty, Himmiche et al. [18] evaluated how uncertain temporal behaviour affects schedule robustness.

Formal and statistical analysis can also characterise distributional or service-level performance beyond a single nominal makespan. Meloni and Pranzo [45] studied makespan quantiles and superquantiles in interval-valued activity networks. Flores-Gómez and Dauzère-Pérès [46] formulated a makespan service-level criterion for flexible job-shop scheduling under machine-related uncertainty, evaluating the probability that makespan remains below a specified threshold.

Existing formal approaches therefore support schedule synthesis, temporal-property verification, robustness analysis, and statistical evaluation of uncertain execution. In most cases, scheduling decisions are generated inside the formal model or supplied as a substantially specified scheduling structure. The use of a learned partial machine order as an intermediate representation for subsequent formal completion remains comparatively underexplored.

2.4. Hybrid Methods

Hybrid methods combine data-driven decision making with explicit formal, logical, or event-based structure. Lassoued and Schwung [47] proposed PetriRL, which integrates Petri nets with event-based deep reinforcement learning for JSSP optimisation: the Petri-net model represents the production process and constraints, while reinforcement learning operates at decision points in the event-driven environment. This provides an example in which the formal/event model is part of the scheduling environment itself.

A complementary pattern appears in safe reinforcement learning. Hamel-De le Court et al. [48] proposed probabilistic shielding, in which formally derived safety information is used to restrict actions that could violate a probabilistic safety requirement. Although this work is not a job-shop scheduling method, it is relevant as evidence that formal analysis can constrain a learned policy rather than merely evaluate its output.

These studies illustrate two representative hybridisation patterns: structured models can define the environment in which learning operates, and formal analysis can restrict the action space of a learned policy. By comparison, reinforcement-learning-based schedulers generally generate decisions directly [14,15,16,17], whereas model-checking-based scheduling approaches perform synthesis or temporal analysis within the formal model [42,43]. RAPO-RL-TAC occupies an intermediate position: learning first reduces the machine-order decision space, and a subsequent formal temporal model completes the deliberately retained relations.

Table 1. Positioning of representative metaheuristic, learning-based, formal, and hybrid scheduling studies.

Main Mechanism	Representative Study	Primary Solution Representation	Formal/Dynamic Completion	Main Focus
Population-based neighbourhood search	Lei [6]	Ordered operation-based representation → complete schedule	None	Specialised neighbourhood search for interval makespan.
Population-based neighbourhood search	Lei [6]	Ordered operation-based representation → complete schedule	None	Specialised neighbourhood search for interval makespan.
Genetic search with interval schedule generation	Díaz et al. [7]	Complete operation/machine sequence	None	Sequence-based optimisation under interval uncertainty.
Elite artificial bee colony	Díaz et al. [21]	Complete machine sequence	None	Elitist interval-makespan optimisation.
Seasonal artificial bee colony	Díaz et al. [22]	Complete machine sequence	None	Seasonal exploration for IJSP optimisation.
Fast elitist artificial bee colony	Díaz et al. [8]	Complete machine sequence	None	Accelerated elitist interval-makespan optimisation.
Two-stage permutable operation groups	Rivière et al. [11]	Partial scheduling decision/permutable groups	Second-stage sequencing after uncertainty realisation	Two-stage robust delayed commitment.
Sequential-flexibility robust scheduling	Ourari et al. [30]	Flexible machine-sequencing representation	Robust adjustment through retained sequential flexibility	Preservation of machine-sequencing flexibility.
GNN + PPOdispatching	Zhang et al. [14]	Incrementally completed schedule	None	Graph-based learned dispatching.
Graph representation + policy learning	Park et al. [15]	Incrementally completed schedule	None	Graph-based learned scheduling.
Deep reinforcement learning with uncertain processing times	Wu et al. [16]	Dynamic scheduling action/rule	Event-driven rescheduling	Learned response to dynamic uncertainty.
Learning under uncertain durations	Infantes et al. [17]	Learned scheduling decisions/robust schedule	No separate formal completion layer	Robust learned scheduling under uncertain durations.
Robustness evaluation under uncertainty	Himmiche et al. [18]	Given schedule + robustness measures	Temporal simulation	Temporal robustness assessment under uncertainty.
Reachability-based production scheduling	Behrmann et al. [42]	Formal model states/task choices	Formal synthesis	Reachability-based scheduling synthesis.
Model checking for flexible manufacturing	Malik and Pena [43]	Formal task sequence	Formal synthesis	Formal scheduling synthesis by model checking.
Petri net + reinforcement learning	Lassoued and Schwung [47]	Event-driven dispatching decisions	Decisions inside formal event model	Reinforcement learning within Petri-net execution semantics.
Probabilistic shielding	Hamel-De le Court et al. [48]	Constrained learned actions	Formal action restriction	Formal restriction of learned actions.

summarises representative studies across the four methodological directions.

3. Problem Definition and Overall Framework

This section formalises the IJSP considered in this study and introduces the information flow of RAPO-RL-TAC.

3.1. Problem Definition

Consider an IJSP instance comprising a job set $\mathcal{J}=\{J_1,\dots ,J_n\}$, a machine set $\mathcal{M}=\{M_1,\dots ,M_m\}$, and an operation set $\mathcal{O}=\{o_1,\dots ,o_N\}$. Each job $J_j$ consists of an ordered sequence of $m_j$ operations that must follow a predefined technological route. Each operation is processed non-preemptively on one designated machine, and the total number of operations is $N={\sum }_{j=1}^n{m}_j$. For the standard $n\times m$ instances considered in this study, each job contains exactly m operations and hence $N=nm$.

For operation $o_i$, let ${\mu }_i\in \mathcal{M}$ denote its designated machine and let its processing time be the closed interval ${\tilde{p}}_i=\left[p_i^{\downarrow },p_i^{\uparrow }\right]$, where $p_i^{\downarrow }$ and $p_i^{\uparrow }$ are the lower and upper processing-time bounds. A feasible schedule assigns a starting time to every operation while satisfying technological precedence, machine mutual exclusion, and non-preemption constraints. Under a realised processing time $p_i\in {\tilde{p}}_i$, the completion time of $o_i$ is $C_i=S_i+p_i$, and the makespan is $C_{max}={max}_{o_i\in \mathcal{O}}{C}_i$. When a complete machine sequence is given, interval starting times, completion times, and makespan can be obtained through interval arithmetic [7]. The midpoint of the resulting interval makespan is used as the benchmark value in the comparative study, whereas the complete temporal behaviour is examined after timed-automata completion.

3.2. Overall Framework

RAPO-RL-TAC is designed as a staged framework that combines reinforcement learning with formal temporal modelling for the IJSP. Instead of requiring a complete machine sequence to be determined before temporal execution, the framework first reduces the machine-conflict decision space through reinforcement learning and then leaves a controlled subset of timing-sensitive relations to be resolved during timed execution. Given an IJSP instance with interval-valued processing times, the overall procedure consists of three successive stages: reinforcement-learning-based partial-order construction, timed-automata-based schedule completion, and statistical analysis of the resulting temporal behaviour. Figure 1 illustrates the information flow among these stages.

In this framework, risk refers to the scheduling risk arising from premature commitment of machine-order relations under interval-induced temporal uncertainty. When the feasible execution windows of two competing operations substantially overlap, their preferred order can be sensitive to the realised processing times; fixing such a relation too early may reduce the temporal flexibility available during execution and lead to unnecessary waiting or an unfavourable completion sequence. RAPO-RL addresses this risk through an interval-derived temporal-determinacy criterion and a machine-wise open-edge budget. Comparatively stable machine conflicts are committed before execution, whereas timing-sensitive conflicts with greater ordering ambiguity are deliberately retained for state-dependent completion by TAC. In this way, risk-aware selective commitment balances the reduction of the machine-order decision space with the preservation of temporal flexibility under interval uncertainty.

(1)

Reinforcement-learning-based partial-order construction. The IJSP instance is first represented as a disjunctive graph containing technological precedence arcs and unresolved same-machine conflict edges. As conflict relations are progressively resolved, this representation evolves into a partial disjunctive graph. A GNN encodes the current graph state by aggregating structural information together with interval-based temporal features, including the propagated lower and upper completion bounds of operations. The resulting node embeddings and graph-level representation provide the state information used by the reinforcement-learning policy. Rather than directly constructing a complete machine sequence, the policy learns which unresolved machine-conflict edge should be processed next. Conflicts with comparatively high temporal determinacy are preferentially committed, whereas a bounded number of temporally ambiguous relations are retained for subsequent completion. After an edge is selected, its feasible direction is determined by the environment according to temporal information and acyclicity-preserving orientation rules. Through repeated interaction and PPO-based policy optimisation, RAPO-RL progressively converts the original conflict graph into a learned machine partial order, thereby reducing the residual combinatorial branching passed to the temporal-execution stage. The learned structure is retained in two complementary forms: the global closure ${\mathbf{P}}^{*}$ records the established reachability relations, while the machine-wise tier matrix $\mathbf{Q}$ provides operational priority information for the next stage.

(2)

Timed-automata-based schedule completion. The learned partial order intentionally leaves a limited number of machine-order relations unresolved because their preferred ordering may depend more strongly on the realised temporal state. These remaining decisions are completed through a network of timed automata. The Job, Controller, and Machine automata use clocks, guards, invariants, synchronisation channels, and request queues to represent technological routes, interval processing durations, machine availability, and competition among operations. The tier matrix $\mathbf{Q}$ is used to guide the selection of jobs that are currently waiting for the same machine, while unresolved alternatives can still be decided according to the evolving execution state. Timed automata therefore transform the learned partial-order representation into complete executable machine sequences without requiring every same-machine relation to be fixed before execution. This stage preserves temporal flexibility while restricting the remaining decision space to the relations deliberately left open by RAPO-RL.

(3)

Statistical analysis of temporal execution. After the residual sequencing decisions have been completed within the timed-automata model, the resulting executions are analysed statistically under interval-valued processing times. Statistical model checking samples admissible timed executions and characterises their completion-time behaviour, including the empirical completion-time distribution, mean estimates, confidence information, and execution variability. This stage complements the conventional interval-makespan midpoint used for benchmark comparison by providing information about how the completed schedules behave across different temporal realisations. The three stages therefore form a sequential learning-to-formal pipeline in which reinforcement learning reduces machine-order uncertainty, timed automata complete the remaining scheduling decisions, and statistical analysis evaluates the temporal behaviour of the resulting executions.

4. Risk-Aware Partial-Order Reinforcement Learning (RAPO-RL)

RAPO-RL is designed to reduce the machine-order decision space before timed execution while preserving a limited number of relations that are strongly affected by interval uncertainty. Starting from the disjunctive-graph representation of an IJSP instance, the method progressively converts selected machine-conflict edges into directed precedence relations. A GNN encodes the evolving partial graph, and a reinforcement-learning policy uses the learned graph representation to determine which unresolved conflict should be processed next. Temporal determinacy and an open-edge budget control which relations are committed during learning and which are retained for subsequent timed-automata completion.

In RAPO-RL, risk awareness refers to explicit sensitivity to interval-induced temporal uncertainty when machine-order relations are committed. The implementation follows three connected stages in this section. First, the risk-aware determinacy assessment in Section 4.3 quantifies the ordering uncertainty of unresolved machine conflicts. Second, the open-edge budget in Section 4.4 retains a controlled number of low-determinacy relations for later completion. Third, the reward design in Section 4.5 incorporates pessimistic residual uncertainty, processing-time interval width, and perturbation sensitivity into policy learning. Together, these mechanisms discourage premature commitment while preserving a bounded degree of temporal flexibility.

The learning policy does not directly determine the orientation of a selected edge. Instead, it learns a global priority over unresolved machine conflicts, while the environment determines the direction of the selected relation using temporal information and acyclicity-preserving rules. This separation allows the GNN and reinforcement-learning policy to focus on identifying informative conflict-resolution decisions without expanding the action space to all possible edge-direction combinations.

Figure 2 provides an overview of this learning-and-resolution workflow and the two learned output representations used by the subsequent completion stage.

4.1. Disjunctive Graph and Partial Disjunctive Graph Representation

An unresolved IJSP instance is represented by a mixed disjunctive graph ${\mathcal{G}}_0=(\mathcal{O},{\mathcal{A}}^c,\mathcal{E})$, where $\mathcal{O}$ is the operation set, ${\mathcal{A}}^c$ contains the directed technological-precedence arcs, and $\mathcal{E}$ contains the undirected conflict edges between operations requiring the same machine. Each edge $\{u,v\}\in \mathcal{E}$ therefore represents a machine-order relation that has not yet been determined.

During RAPO-RL, selected conflict edges are progressively oriented. At decision step t, the current partial disjunctive graph is represented as

$${\mathcal{G}}_t=\left(\mathcal{O},{\mathcal{A}}^c\cup {\mathcal{D}}_t^{\mathrm{dir}},{\mathcal{E}}_t^{\mathrm{open}}\right).$$

(1)

where ${\mathcal{D}}_t^{\mathrm{dir}}$ is the set of machine-conflict relations already oriented at step t, and ${\mathcal{E}}_t^{\mathrm{open}}$ is the set of unresolved machine-conflict edges. Initially, ${\mathcal{D}}_0^{\mathrm{dir}}=⌀$ and ${\mathcal{E}}_0^{\mathrm{open}}=\mathcal{E}$. Unlike complete-sequence construction, ${\mathcal{E}}_T^{\mathrm{open}}$ is allowed to remain non-empty at termination because selected timing-sensitive relations are deliberately retained for the downstream completion stage.

The directed part of ${\mathcal{G}}_t$, consisting of technological arcs and currently established machine-precedence relations, provides the structural input to the GNN. Open conflict edges are maintained separately as candidate decisions rather than being inserted into the message-passing adjacency structure. This avoids propagating messages through dense unresolved machine-conflict cliques and allows the encoded graph to represent only precedence information that has already been established [14].

Because ${\mathcal{G}}_t$ is generally incomplete, it does not define a unique executable schedule. Instead, lower and upper completion-time estimates are propagated along the currently established directed relations:

$$\begin{array}{cc}\hfill C_i^{\downarrow }& =p_i^{\downarrow }+\underset{j\in {\mathrm{Pred}}_t\left(i\right)}{max}{C}_j^{\downarrow },\hfill \\ \hfill C_i^{\uparrow }& =p_i^{\uparrow }+\underset{j\in {\mathrm{Pred}}_t\left(i\right)}{max}{C}_j^{\uparrow }.\hfill \end{array}$$

(2)

where $C_i^{\downarrow }$ and $C_i^{\uparrow }$ are the lower and upper completion estimates of operation $o_i$, respectively; $p_i^{\downarrow }$ and $p_i^{\uparrow }$ are its processing-time bounds; and ${\mathrm{Pred}}_t\left(i\right)$ is the set of currently established predecessors of $o_i$. The maximum is taken as zero when ${\mathrm{Pred}}_t\left(i\right)=⌀$. These quantities do not represent the final makespan; they provide temporal features for evaluating and learning from the evolving partial order.

4.2. RAPO-RL-Based Partial-Order Learning

The partial-order construction process is formulated as a sequential decision problem. At each decision step, RAPO-RL receives the current partial disjunctive graph, encodes its structural and temporal state using a GNN, evaluates the currently admissible machine-conflict edges, and selects one edge for resolution. The environment subsequently determines its direction and updates the graph. Repeating this interaction gradually converts the unresolved disjunctive graph into a machine partial order.

The information flow is therefore: partial disjunctive graph and node features → GNN state encoder → node and graph embeddings → edge-level actor policy → selected open conflict edge → environment update → updated partial graph.

Figure 3 illustrates how the scheduling state is converted from graph and node-feature inputs into learned embeddings, edge-selection probabilities, and the next partial-graph state.

4.2.1. Markov Decision Process

The partial-order construction process is modelled as a Markov decision process $\mathcal{M}=\langle \mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma \rangle$ [14]. At decision step t, the state $s_t$ contains the current partial graph together with its temporal and structural features. An action selects one currently admissible machine-conflict edge for processing. Importantly, the action identifies which conflict should be resolved; it does not specify whether $u\to v$ or $v\to u$ should be introduced. That direction is determined by the environment after the edge has been selected. The transition to $s_{t+1}$ therefore reflects both the learned edge choice and the environment-side orientation rule.

4.2.2. State Representation and Node Features

The GNN input combines the current directed precedence structure with interval-derived temporal information. Each operation $o_i$ is represented by the node-feature vector

$${\mathbf{x}}_i^{\left(t\right)}={\left[{\displaystyle \frac{C_i^{\downarrow }}{\eta }},{\displaystyle \frac{C_i^{\uparrow }}{\eta }},{\delta }_i^{\left(t\right)}\right]}^{\top }.$$

(3)

where $C_i^{\downarrow }$ and $C_i^{\uparrow }$ are the propagated lower and upper completion estimates, $\eta =1000$ is the time-normalisation constant, and ${\delta }_i^{\left(t\right)}\in [0,1]$ is the proportion of candidate same-machine conflict edges incident to $o_i$ that have already been oriented. The first two components describe the current optimistic and pessimistic temporal position of the operation, while the third represents local progress in partial-order construction.

The GNN adjacency contains only established technological and machine-precedence relations. An incoming neighbour of $o_i$ therefore corresponds to an operation that is currently known to precede $o_i$. Open machine-conflict edges remain outside the adjacency and are handled later by the edge-selection policy and its action mask.

4.2.3. GNN State Encoder

The purpose of the GNN is to transform the partially resolved scheduling graph into learned representations that capture both local operation relationships and the global state of the current partial order. The initial node embedding is ${\mathbf{h}}_i^{\left(0\right)}={\mathbf{x}}_i^{\left(t\right)}$. Using sum aggregation, each message-passing layer applies a multilayer perceptron (MLP), batch normalisation (BN), and a rectified linear unit (ReLU) activation to update operation $o_i$ as

$${\mathbf{h}}_i^{(l+1)}=\mathrm{ReLU}\left[\mathrm{BN}\left({\mathrm{MLP}}^{\left(l\right)}\left({\mathbf{h}}_i^{\left(l\right)}+\sum _{j\in {\mathrm{Pred}}_t\left(i\right)}{\mathbf{h}}_j^{\left(l\right)}\right)\right)\right].$$

(4)

where ${\mathbf{h}}_i^{\left(l\right)}$ is the embedding of operation $o_i$ at layer l, ${\mathrm{Pred}}_t\left(i\right)$ is its predecessor set in the current directed graph, and ${\mathrm{MLP}}^{\left(l\right)}$ denotes the trainable MLP at layer l. The trainable weight matrices and bias vectors inside the MLP are shared across operation nodes within the same layer and are learned jointly with the reinforcement-learning policy through PPO; they are not manually assigned scheduling weights.

After the final message-passing layer, the node embeddings are pooled into a graph-level representation:

$${\mathbf{h}}_G={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{\mathbf{h}}_i.$$

(5)

where N is the number of operations, ${\mathbf{h}}_i$ is the final embedding of operation $o_i$, and ${\mathbf{h}}_G$ summarises the overall partial-order state. Thus, the GNN produces operation-level embeddings for local conflict evaluation and a graph-level embedding for global context.

4.2.4. Edge-Level Actor Network

For each admissible unresolved machine-conflict edge $e=\{u,v\}$, the actor combines the embeddings of the two endpoint operations with the global graph representation. The edge representation and its policy score are defined by

$$\begin{array}{cc}\hfill {\mathit{\varphi }}_e& =\left[{\mathbf{h}}_u\Vert {\mathbf{h}}_v\Vert {\mathbf{h}}_G\right],\hfill \\ \hfill z_e& ={\mathbf{W}}_2\tanh \left({\mathbf{W}}_1{\mathit{\varphi }}_e+{\mathbf{b}}_1\right)+b_2.\hfill \end{array}$$

(6)

where ‖ denotes vector concatenation; ${\mathbf{W}}_1$, ${\mathbf{W}}_2$, ${\mathbf{b}}_1$, and $b_2$ are trainable actor parameters; and $z_e$ is the scalar score assigned to edge e. The same actor weights are shared across all candidate edges, allowing conflicts from different machines and decision steps to be evaluated by one learned policy.

After prohibited edges are masked, the remaining scores are normalised into the edge-selection distribution

$${\pi }_{\theta }(a_t=e\mid s_t)={\displaystyle \frac{\exp \left(z_e\right)}{{\sum }_{e^{\prime }\in {\mathcal{S}}^{\left(t\right)}}\exp \left(z_{e^{\prime }}\right)}},e\in {\mathcal{S}}^{\left(t\right)}.$$

(7)

where ${\pi }_{\theta }$ is the actor policy with parameters $\theta$, ${\mathcal{S}}^{\left(t\right)}$ is the current admissible-edge set, and $a_t=e$ denotes selection of edge e. The actor therefore answers the question: which unresolved machine conflict should be processed next, given the current partial-order state?

4.2.5. Critic Network

A critic network operates in parallel on the graph-level embedding ${\mathbf{h}}_G$ and estimates the value of the current state, $V_{\varphi }\left(s_t\right)={\mathrm{MLP}}_V\left({\mathbf{h}}_G\right)$. The actor and critic share the GNN encoder but use separate output heads, so the encoder is trained to support both edge selection and state-value estimation. The critic is used only as the baseline required by actor–critic optimisation and does not introduce an additional scheduling decision.

4.2.6. PPO Policy Optimisation

The GNN encoder, actor, and critic are trained jointly using PPO [14]. PPO updates the policy from collected trajectories while clipping excessively large policy changes between successive updates. Advantages are computed from the standardised trajectory returns and the critic baseline. The optimisation objective combines the clipped policy loss, value-function loss, and entropy regularisation, i.e., $L=c_p{L}_{\mathrm{policy}}+c_v{L}_{\mathrm{value}}+c_e{L}_{\mathrm{ent}}$. Because PPO is used as the optimisation mechanism rather than as a new methodological contribution, the standard component losses are not expanded further here. This keeps the presentation focused on the scheduling-specific elements: graph-state encoding, edge prioritisation, selective commitment, and environment-side orientation.

4.3. Risk-Aware Determinacy Assessment for Selective Commitment

Risk-aware selective commitment in RAPO-RL begins by assessing the uncertainty associated with each unresolved machine conflict. Under interval processing times, the possible execution periods of two same-machine operations may overlap to different degrees, so some conflicts exhibit a comparatively stable ordering preference whereas others remain sensitive to the realised processing times. RAPO-RL quantifies this ordering uncertainty through a time-conflict determinacy score, which provides the primary criterion for deciding whether a conflict should be committed during learning or retained for later completion.

4.3.1. Estimated Execution Window

For each operation $o_i$, an estimated execution window is constructed from its current completion bounds:

$$W_i=[s_i,c_i]=\left[C_i^{\downarrow }-p_i^{\downarrow },C_i^{\uparrow }\right].$$

(8)

where $s_i$ is the estimated earliest start time, $c_i$ is the pessimistic completion estimate, $C_i^{\downarrow }$ and $C_i^{\uparrow }$ are the propagated completion bounds, and $p_i^{\downarrow }$ is the lower processing-time bound. The numerical window width is $w_i=max({10}^{-6},c_i-s_i)$.

4.3.2. Normalised Temporal Overlap

For an unresolved same-machine conflict $\{u,v\}$, the normalised overlap between the two estimated execution windows is

$${\mathrm{overlap}}_{uv}=min\left(1,{\displaystyle \frac{max\left(0,min(c_u,c_v)-max(s_u,s_v)\right)}{min(w_u,w_v)}}\right).$$

(9)

where $s_u,s_v$ and $c_u,c_v$ are the estimated start and completion limits of operations u and v, and $w_u,w_v$ are their corresponding window widths. Values near zero indicate well-separated execution windows, whereas values near one indicate strong temporal overlap.

4.3.3. Ordering Criticality

Temporal overlap alone does not indicate how important the ordering choice is to the current schedule. We therefore compare the pessimistic local completion estimates associated with the two possible orders. Their normalised ordering criticality is

$${\mathrm{crit}}_{uv}=min\left(1,{\displaystyle \frac{max\left(C_u^{\uparrow }+p_v^{\uparrow },C_v^{\uparrow }+p_u^{\uparrow }\right)}{C_{max}^{\uparrow }}}\right).$$

(10)

where $C_u^{\uparrow }$ and $C_v^{\uparrow }$ are the current pessimistic completion estimates, $p_u^{\uparrow }$ and $p_v^{\uparrow }$ are the corresponding upper processing-time bounds, and $C_{max}^{\uparrow }={max}_i{C}_i^{\uparrow }$ is the current global pessimistic completion estimate.

4.3.4. Time-Conflict Determinacy Score

The final time-conflict determinacy score combines temporal overlap and ordering criticality:

$${\kappa }_{uv}=1-{\mathrm{overlap}}_{uv}{\mathrm{crit}}_{uv},{\kappa }_{uv}\in [0,1].$$

(11)

where ${\mathrm{overlap}}_{uv}$ measures the temporal overlap between the two operations and ${\mathrm{crit}}_{uv}$ measures the potential importance of their ordering. A larger ${\kappa }_{uv}$ indicates a conflict whose ordering is comparatively more determinate under the current temporal approximation, whereas a smaller value identifies a more temporally ambiguous relation. Given the determinacy threshold $\tau$, conflicts with ${\kappa }_{uv}\ge \tau$ are eligible for learning-stage resolution, whereas conflicts with ${\kappa }_{uv}<\tau$ are treated as low-determinacy candidates for retention, subject to the machine-wise budget in Section 4.4.

4.4. Open-Edge Budget Mechanism

The open-edge budget forms the second component of risk-aware selective commitment. A low determinacy score indicates that a relation may benefit from being resolved later during timed execution; however, retaining every such relation would leave an unnecessarily large residual decision space for the downstream timed-automata model. RAPO-RL therefore introduces a machine-wise open-edge budget.

For each machine, low-determinacy conflicts are ranked by $\kappa$. When their number does not exceed the prescribed budget $cap$, they are retained as open relations. If the number exceeds the budget, only the $cap$ conflicts with the smallest determinacy scores are retained; the remaining conflicts are returned to the admissible action set and must be processed by RAPO-RL. Consequently, the terminal partial order satisfies

$$\left|{\mathcal{W}}_k^{\left(T\right)}\right|\le cap,k=1,\dots ,m.$$

(12)

where ${\mathcal{W}}_k^{\left(T\right)}$ is the set of low-determinacy conflicts remaining open on machine $M_k$ at the end of learning, $cap$ is the maximum number of such relations allowed per machine, and m is the number of machines. This mechanism creates an explicit trade-off between early conflict reduction and retained temporal flexibility: relations with relatively clear temporal structure are resolved before timed execution, while only a controlled number of the most temporally ambiguous relations remain open.

4.5. Environment-Side Orientation Rules and Reward Design

The reinforcement-learning policy determines which unresolved edge should be processed, whereas the environment determines the direction assigned to that edge. The environment also provides the reward used to train the policy. This separation keeps the learned action space compact while enforcing precedence consistency independently of the policy. The reward design further embeds risk awareness into policy learning by penalising decisions that increase pessimistic residual uncertainty or prematurely commit relations associated with wide processing-time intervals.

4.5.1. Local Edge Orientation Rules

To evaluate candidate orientations, the current partial graph is associated with the relaxed completion proxy

$${\mathsf{\Phi }}_{\mathrm{rob}}\left({\mathcal{G}}_t\right)={\displaystyle \frac{1}{2}}\left(\underset{i}{max}{C}_i^{\downarrow }+\underset{i}{max}{C}_i^{\uparrow }\right).$$

(13)

where ${\mathcal{G}}_t$ is the current partial graph and $C_i^{\downarrow },C_i^{\uparrow }$ are the propagated completion bounds of operation $o_i$. The quantity ${\mathsf{\Phi }}_{\mathrm{rob}}$ is used only to rank local decisions and should not be interpreted as the realised or expected makespan of the completed schedule.

For a selected edge $\{u,v\}$, directions that would introduce a directed cycle are discarded. If the estimated execution windows are strictly separated, the temporally consistent acyclic direction is adopted directly. Otherwise, the environment selects

$$d_{u,v}^{*}=\arg \underset{d\in {\mathcal{D}}_{u,v}^{\left(t\right)}}{min}{\mathsf{\Phi }}_{\mathrm{rob}}\left({\mathcal{G}}_t\cup \left\{d\right\}\right).$$

(14)

where ${\mathcal{D}}_{u,v}^{\left(t\right)}$ is the set of acyclic candidate directions for the selected edge and $d_{u,v}^{*}$ is the direction producing the smallest relaxed completion proxy. This division allows the policy to focus on global edge prioritisation while the environment uses temporal structure and completion bounds for local orientation.

4.5.2. Pessimistic Reserve Metric

The immediate effect of an orientation is complemented by a pessimistic reserve that reflects the remaining uncertainty associated with unresolved conflicts:

$$B_{\mathrm{pess}}\left(s_t\right)=C_{max}^{\uparrow }\left(s_t\right)+\sum _{\{u,v\}\in {\mathcal{E}}_t^{\mathrm{open}}}\left|\left(C_u^{\uparrow }+p_v^{\uparrow }\right)-\left(C_v^{\uparrow }+p_u^{\uparrow }\right)\right|.$$

(15)

where $s_t$ denotes the current state, $C_{max}^{\uparrow }\left(s_t\right)$ is its maximum upper completion estimate, ${\mathcal{E}}_t^{\mathrm{open}}$ is the unresolved conflict set, and the summation measures the difference between the two local pessimistic ordering estimates of each open conflict.

4.5.3. Step-Wise Reward

For the selected edge $\{u,v\}$, let $w_{uv}=(p_u^{\uparrow }-p_u^{\downarrow })+(p_v^{\uparrow }-p_v^{\downarrow })$ denote the combined processing-time interval width. The reward associated with an accepted conflict resolution is

$$\begin{array}{cc}\hfill R_t=& {\mathsf{\Phi }}_{\mathrm{rob}}\left({\mathcal{G}}_t\right)-{\mathsf{\Phi }}_{\mathrm{rob}}\left({\mathcal{G}}_{t+1}\right)\hfill \\ \hfill & -{\lambda }_p\left[B_{\mathrm{pess}}\left(s_{t+1}\right)-B_{\mathrm{pess}}\left(s_t\right)\right]-{\lambda }_w{w}_{uv}.\hfill \end{array}$$

(16)

where $R_t$ is the reward for the transition from $s_t$ to $s_{t+1}$, ${\lambda }_p=0.05$ and ${\lambda }_w=0.01$ are weighting coefficients, $B_{\mathrm{pess}}$ is the pessimistic reserve, and $w_{uv}$ is the interval-width term defined above. The first term rewards improvement of the relaxed completion proxy, the second discourages decisions that increase residual pessimistic uncertainty, and the third discourages premature commitment of relations involving wide processing-time intervals.

4.5.4. Termination Robustness Reward

A terminal robustness term complements the step-wise reward by evaluating the learned partial order under sampled processing-time perturbations. Let ${\overline{C}}_{max}$ denote the makespan obtained using interval-midpoint processing times and let $C_{max}^{\left(k\right)}$ be the corresponding makespan under Monte Carlo sample k. The relative-deviation measure is

$$\epsilon ={\displaystyle \frac{1}{K_{\mathrm{MC}}}}{\displaystyle \sum _{k=1}^{K_{\mathrm{MC}}}}{\displaystyle \frac{\left|C_{max}^{\left(k\right)}-{\overline{C}}_{max}\right|}{{\overline{C}}_{max}}}.$$

(17)

where $K_{\mathrm{MC}}$ is the number of terminal Monte Carlo samples, ${\overline{C}}_{max}$ is the midpoint-based reference makespan, and $C_{max}^{\left(k\right)}$ is the makespan obtained from sample k. The terminal reward is $R_{\mathrm{term}}=-\beta \epsilon$, where $\beta =200$ controls the contribution of robustness to the learning signal. A smaller variation under processing-time perturbations therefore produces a larger terminal reward.

4.6. Training Procedure of RAPO-RL

Each RAPO-RL episode starts from the unresolved disjunctive graph of one IJSP instance and progressively constructs a partial order. At every decision step, the current completion bounds and node features are updated, the GNN generates node and graph embeddings, and the actor evaluates the admissible machine-conflict edges. After an edge is selected, the environment determines its orientation, updates the graph, and returns the corresponding reward. The next decision is therefore based on a newly encoded partial-order state.

Training is performed using multiple parallel environments and PPO updates. The GNN encoder, actor, and critic are trained jointly so that the learned graph representations become informative for edge-selection decisions. An episode terminates when no admissible conflict remains or the prescribed maximum number of decision steps is reached. Algorithm 1 summarises the complete training procedure.

Algorithm 1 Compact pseudo-code of RAPO-RL partial-order learning and training.

Input: ${\mathcal{G}}_0=(\mathcal{O},{\mathcal{A}}^c,\mathcal{E})$, $\tau$, $cap$, $T_{max}$, $N_{\mathrm{env}}$, $Q_{max}=10000$, PPO parameters $\theta ,\varphi$. Output: trained policy ${\pi }_{\theta }$ and instance-level outputs $({\mathcal{D}}_T^{\mathrm{dir}},{\mathcal{E}}_T^{\mathrm{open}},{\mathbf{P}}^{*},\mathbf{Q})$.   1: Initialise encoder, actor, critic, and optimiser states.   2: for  $q=1,\dots ,Q_{max}$  do   3:     Reset $N_{\mathrm{env}}$ environments with ${\mathcal{G}}_0$.   4:     while active environment exists do   5:         Compute $C^{\downarrow },C^{\uparrow }$, node features $X_t$, and determinacy scores ${\kappa }^{\left(t\right)}$.   6:         Construct the action mask $m_e^{\left(t\right)}=⊮[e\notin {\mathcal{S}}^{\left(t\right)}]$ subject to $\tau$ and $cap$.   7:         Encode $H_t=\mathrm{GNN}({\mathcal{G}}_t,X_t)$ and sample $a_t\sim {\pi }_{\theta }(·\mid H_t,m^{\left(t\right)})$.   8:         Apply $d_t\leftarrow \mathrm{Orient}(a_t;{\kappa }^{\left(t\right)},{\mathsf{\Phi }}_{\mathrm{rob}},\mathrm{acyclicity})$ and update ${\mathcal{G}}_{t+1}$.   9:         Store $(s_t,a_t,r_t,\log {\pi }_{\theta }(a_t\mid s_t),V_{\varphi }\left(s_t\right))$. 10:         If terminal, add $R_{\mathrm{term}}=-\beta \epsilon$ with $\epsilon$ estimated from Monte Carlo completion samples. 11:     end while 12:     Compute standardised returns ${\widehat{G}}_t$ and advantages $A_t={\widehat{G}}_t-V_{\varphi }\left(s_t\right)$. 13:     Update $(\theta ,\varphi )$ using the PPO objective defined in Section 4.2.6. 14:     Every 100 iterations, evaluate the greedy policy and keep the best checkpoint. 15: end for 16: output ${\pi }_{\theta }$ and $({\mathcal{D}}_T^{\mathrm{dir}},{\mathcal{E}}_T^{\mathrm{open}},{\mathbf{P}}^{*},\mathbf{Q})$.

Algorithm 1 uses compact notation for the training loop. The input statement specifies the initial disjunctive graph ${\mathcal{G}}_0$, determinacy threshold $\tau$, per-machine open-edge budget $cap$, maximum episode length $T_{max}$, number of parallel environments $N_{\mathrm{env}}$, maximum number of PPO update iterations $Q_{max}$, and actor–critic parameters $\theta$ and $\varphi$. The set ${\mathcal{S}}^{\left(t\right)}$ in Step 6 denotes the current conflicts protected from policy selection because they are retained by the low-determinacy rule and the machine-wise open-edge budget; $⊮[·]$ is the indicator function, which equals 1 when its argument is true and 0 otherwise, so only conflicts outside ${\mathcal{S}}^{\left(t\right)}$ remain selectable. The output statement records the trained policy ${\pi }_{\theta }$ together with the final directed and open conflict sets and the two instance-level representations ${\mathbf{P}}^{*}$ and $\mathbf{Q}$.

The numbered algorithm steps operate as follows. Step 1 initialises the shared GNN encoder, actor, critic, and optimiser states. Step 2 starts the outer PPO-update loop indexed by q. Step 3 resets the $N_{\mathrm{env}}$ parallel environments to ${\mathcal{G}}_0$. Step 4 opens the within-episode loop for all environments that have not terminated. Step 5 recomputes the lower and upper completion estimates, node-feature matrix $X_t$, and determinacy scores ${\kappa }^{\left(t\right)}$ from the current partial graphs. Step 6 converts the determinacy threshold and open-edge budget into the binary action mask $m^{\left(t\right)}$, preventing protected low-determinacy conflicts from being selected. Step 7 applies the GNN to obtain the encoded state $H_t$ and samples the next admissible conflict $a_t$ from ${\pi }_{\theta }$. Step 8 invokes the environment-side orientation rule, which uses ${\kappa }^{\left(t\right)}$, the relaxed completion proxy ${\mathsf{\Phi }}_{\mathrm{rob}}$, and acyclicity to orient the selected conflict and construct ${\mathcal{G}}_{t+1}$. Step 9 stores the transition, log-probability, reward, and critic value required for the PPO update. Step 10 adds the terminal robustness reward $R_{\mathrm{term}}=-\beta \epsilon$ when an episode ends, where $\epsilon$ is estimated from Monte Carlo completion samples. Step 11 closes the within-episode loop after all active environments have terminated or reached $T_{max}$. Step 12 converts the collected trajectories into standardised returns ${\widehat{G}}_t$ and advantages $A_t$. Step 13 updates the actor–critic parameters with the PPO objective described in Section 4.2.6. Step 14 performs periodic greedy evaluation and retains the best checkpoint. Step 15 closes the outer training loop after $Q_{max}$ updates. Finally, Step 16 returns the trained policy and the learned partial-order outputs used by TAC.

4.7. Partial-Order Matrix Generation

The terminal partial disjunctive graph contains both established machine-precedence relations and a bounded number of unresolved conflicts. To support subsequent formal execution, the learned structure is converted into two complementary representations: the global closure ${\mathbf{P}}^{*}$, which preserves the structural reachability information learned by RAPO-RL, and the machine-wise tier matrix $\mathbf{Q}$, which provides the operational priority structure used during timed-automata completion.

4.7.1. Global Priority Transitive Closure

A dummy source S and dummy sink T are added to the operation graph, together with the technological and learned machine-precedence relations. The transitive closure of this directed graph defines ${\mathbf{P}}^{*}$. An entry $P_{ij}^{*}=1$ indicates that a directed path from $o_i$ to $o_j$ has already been established before timed execution. The global closure therefore records the structural consequences of the learned partial order and supports consistency and traceability analysis. It is not used to force every reachable relation as a hard execution prerequisite in the standard timed-automata execution model.

4.7.2. Machine-Wise Priority Matrix via Antichain Layering

For each machine, the learned local precedence graph is decomposed into successive antichain tiers using topological layering. Operations with no unresolved predecessor in the current residual graph are assigned to the same tier, removed, and the process is repeated until all operations on the machine have been assigned. Operations in the same tier remain mutually incomparable and therefore preserve their unresolved relative order. A compression step may move an incomparable operation to an earlier tier when all of its established predecessors already lie in preceding tiers; this preserves every learned precedence relation while avoiding artificial tier separation.

The resulting machine-wise matrix is defined by

$$Q_{k,j}=\mathrm{tier}\left(o_{j,k}\right).$$

(18)

where $Q_{k,j}$ is the tier assigned to the operation of job $J_j$ processed on machine $M_k$, $o_{j,k}$ denotes that operation, and $\mathrm{tier}(·)$ is the antichain layer obtained from the learned machine partial order. Smaller tier values represent earlier learned priority, whereas equal-tier operations remain available for online resolution. The two outputs consequently play different roles: ${\mathbf{P}}^{*}$ preserves the global structural record of the learned partial order, whereas $\mathbf{Q}$ provides a compact operational representation for timed execution.

5. Timed Automata Completion (TAC)

TAC converts the partial-order output of RAPO-RL into executable schedules under interval processing times while explicitly modelling synchronisation, queue evolution, and duration uncertainty.

5.1. Timed Automata Model Structure

The TAC model represents residual machine-sequence uncertainty and continuous processing-time uncertainty through three reusable timed-automata templates: Job, Machine, and Controller. For a generic n-job, m-machine instance, the network contains one Job automaton per job, one Machine automaton per machine, and one Controller. The Job template is parameterised by $\mathtt{Jid}\_\mathtt{t}jid$, the Machine template by $\mathtt{Mid}\_\mathtt{t}mid$, and the Controller carries no formal parameter. Shared data provide interval bounds and job routes; the Controller maintains request queues, and the Machine template consults $\mathbf{Q}$ when selecting among currently waiting requests. Figure 4 summarises the interaction among the three templates, while Section 5.2, Section 5.3 and Section 5.4 detail their transition semantics.

To implement the timed-automata representation in the proposed framework, we use the UPPAAL Statistical Model Checking (UPPAAL-SMC) tool as the modelling, simulation, and statistical-verification environment [40,41]. Each job, machine, and residual sequencing decision is represented through timed-automata templates whose clocks, guards, invariants, and synchronisation channels describe temporal evolution and resource interactions. The partial-order information learned by RAPO-RL is transferred to TAC primarily through the tier matrix $\mathbf{Q}$, which guides queue-level machine selection rather than imposing every learned relation as a global hard execution prerequisite; same-tier conflicts remain available for online completion. This implementation supports statistical model checking of sampled timed executions and analysis of completion-time distributions under interval processing-time uncertainty.

5.2. Job Automaton

Each job is represented by a parameterised automaton with the principal locations Ready, Wait, Processed, and Finished. From Ready, a job requests the machine required by its next operation and enters Wait. Once admitted by the corresponding Machine automaton, it synchronises into Processed and resets its local operation clock to zero.

Let x be that local clock and let the current operation have duration interval $\left[p_{jr}^{\downarrow },p_{jr}^{\uparrow }\right]$. The Processed location uses the invariant $x\le p_{jr}^{\uparrow }$, and its completion transition uses the guard $x\ge p_{jr}^{\downarrow }$. Thus, the operation may complete at any time satisfying $x\in \left[p_{jr}^{\downarrow },p_{jr}^{\uparrow }\right]$. After completion, the job releases the machine, advances its operation index, and returns to Ready; after all operations have been processed, it enters Finished.

The state transitions and timing constraints of the Job automaton are illustrated in Figure 5.

5.3. Controller Automaton

The Controller receives the machine requests emitted by the Job automata and inserts them into the corresponding waiting queues. Its self-loop uses $\mathtt{select}e:Jid\_t,h:Mid\_t$ together with the synchronisation $\mathtt{request}\left[e\right]\left[h\right]?$ to receive any currently enabled job-machine request. This mechanism represents the possible interleavings when several jobs issue requests concurrently.

Figure 6 shows the Controller template and its request-handling transition structure.

5.4. Machine Automaton

Each machine is represented by an automaton whose operational locations are Idle, a committed selection location C, and Busy. When machine $M_k$ is idle and its queue is non-empty, let ${\mathcal{Q}}_k$ denote the set of jobs currently waiting for that machine. The automaton identifies the minimum priority tier $q_{min}^{\left(k\right)}={min}_{j\in {\mathcal{Q}}_k}{Q}_{k,j}$ and forms the candidate set

$${\mathcal{J}}_{min}^{\left(k\right)}=\left\{j\in {\mathcal{Q}}_k\mid Q_{k,j}=q_{min}^{\left(k\right)}\right\}.$$

(19)

UPPAAL selects a candidate through $\mathtt{select}j:Jid\_t$ with the guard $j\in {\mathcal{Q}}_k\wedge Q_{k,j}=q_{min}^{\left(k\right)}$. The committed location prevents time from elapsing during machine assignment and therefore avoids artificial waiting.

The corresponding machine-selection and processing transitions are shown in Figure 7.

5.5. Guided Nondeterministic Completion

The tier matrix guides rather than hard-constrains the UPPAAL selection mechanism.

If $Q_{k,j_1}<Q_{k,j_2}$, the lower-tier operation is preferred when both requests are currently waiting. The comparison is evaluated only over the current queue: a higher-tier operation is therefore not blocked solely because a lower-tier operation has not yet entered the queue. This queue-level priority semantics allows the timed-automata model to adapt the realised machine sequence to the evolving temporal state.

If $Q_{k,j_1}=Q_{k,j_2}$, the two operations belong to the same tier and their relative order remains unresolved. When several same-tier jobs are simultaneously available, $\left|{\mathcal{J}}_{min}^{\left(k\right)}\right|>1$, the select construct generates alternative legal execution branches. These branches complete the machine-sequencing alternatives intentionally retained by RAPO-RL.

The resulting execution semantics preserves flexibility in two ways. Same-tier jobs retain an explicit nondeterministic choice, while queue availability can also make a currently available higher-tier job execute before a lower-tier job that has not yet entered the queue. Thus, $\mathbf{Q}$ guides the realised machine sequence without imposing machine idling or a global hard execution order.

5.6. Statistical Model Checking

For the job-shop models considered here, exhaustive state-space exploration may become computationally prohibitive as residual branching grows. The final timed-automata model is therefore assessed with UPPAAL-SMC, which samples stochastic timed behaviour without exhaustively enumerating the complete state space.

Let t denote the global observation clock used for the completion-time analysis. For a specified time bound T, the principal time-bounded completion query is

$$\mathrm{Pr}[t\le T]\left(\diamond {\displaystyle \underset{j=1}{\overset{n}{\bigwedge }}}{J}_j.\mathrm{Finished}\right).$$

(20)

UPPAAL-SMC estimates the probability that all jobs complete within the specified bound and also returns sampled completion-time behaviour that can be summarised by probability-density outputs, sample means, and confidence information.

The resulting statistics reflect both the discrete choices retained in the machine partial order and the continuous variation of processing times within their intervals. These results are model-based estimates under the UPPAAL-SMC stochastic semantics, with uncertainty quantified by the reported confidence information; they do not represent a uniform aggregation over all nondeterministic execution paths.

5.7. Monte Carlo Robustness Evaluation

Monte Carlo sampling is used at two distinct stages. During RAPO-RL training, a small sample set provides the terminal robustness reward defined in Section 4.5.4. After the learned tier matrix $\mathbf{Q}$ is transferred to TAC, UPPAAL-SMC independently evaluates the completed timed model and reports completion-time and time-bounded statistics. The interval-makespan midpoint remains the nominal criterion used for comparison with established IJSP methods.

6. Experiments and Results

The experiments evaluate seven aspects of RAPO-RL-TAC: nominal schedule quality under the conventional midpoint criterion, statistical significance across matched benchmark instances, robustness to PPO/GNN training randomness, computational and structural efficiency, the contributions of partial-order learning and queue-level completion, sensitivity to the determinacy threshold and open-edge budget, and the additional temporal information provided by UPPAAL-SMC. These aspects are examined through benchmark comparison with paired significance testing, random-seed robustness analysis, computational profiling and residual-conflict analysis, controlled ablation, hyperparameter sensitivity analysis, and statistical model checking.

6.1. Experimental Setup and Evaluation Procedure

RAPO-RL is evaluated as an instance-specific optimiser, with a dedicated policy trained independently for each benchmark instance. Accordingly, the experiments assess per-instance optimisation rather than cross-instance generalisation. For each instance, training uses 10,000 PPO update iterations, $N_{\mathrm{env}}=10$ parallel environments, $T_{max}=2000$, the determinacy threshold $\tau =0.05$, and $K_{\mathrm{MC}}=8$ Monte Carlo samples for terminal robustness evaluation. Every 100 updates, deterministic checkpoint evaluation is performed for the current instance and the best checkpoint is retained; inference then uses greedy edge selection. Processing-time interval bounds remain fixed throughout optimisation and evaluation. The open-edge budget is fixed a priori to $cap=3$ for every reported RAPO-RL-TAC run and is not tuned per benchmark instance. All experiments were conducted under Windows 11 with 16 GB of random-access memory (RAM), using Python 3.10.5, PyTorch 2.11.0, CUDA 12.6, and UPPAAL 5.0.0.

After greedy inference, ${\mathbf{P}}^{*}$ and $\mathbf{Q}$ are fixed and the TAC stage is run independently 30 times for each instance. Each exported .uctr trace is decoded from Machine selection events into a complete per-machine permutation ${\pi }_r$. Holding this permutation fixed, the makespan is evaluated once at the lower processing-time bounds and once at the upper bounds, yielding

$$M_r={\displaystyle \frac{1}{2}}\left[C_{max}({\pi }_r;{\mathbf{p}}^{\downarrow })+C_{max}({\pi }_r;{\mathbf{p}}^{\uparrow })\right],\overline{M}={\displaystyle \frac{1}{30}}{\displaystyle \sum _{r=1}^{30}}{M}_r.$$

(21)

Table 2. Comparison of midpoint makespan values on the selected IJSP benchmark instances. Lower is better. The RAPO-RL-TAC column reports the mean midpoint over 30 independently completed UPPAAL traces for the fixed learned representation of each instance. The reference lower bound (LB) column reports the listed bound. PNS and GA are literature comparison values; fEABC and fEABC-LS are results from our reimplementation following the cited fEABC method.

Instance	LB	PNS	GA	fEABC	fEABC-LS	RAPO-RL-TAC
ORB1	1059	1187.00	1171.12	1158.03	1172.10	1162.73
ORB2	888	968.25	971.93	941.07	931.07	942.40
ORB3	1005	1145.23	1117.23	1105.20	1031.07	1094.47
ORB4	1005	1110.85	1087.13	1052.07	1035.07	1080.00
ORB5	887	974.60	955.02	943.07	944.07	996.20
LA16	945	1019.40	1024.52	982.10	960.03	973.43
LA17	784	834.45	834.50	790.37	793.47	811.03
LA18	848	912.95	900.75	861.40	861.40	902.07
LA19	842	919.65	904.95	896.57	867.27	893.77
LA20	902	966.50	952.87	926.57	914.50	928.93
LA21	1046	1173.45	1150.73	1117.72	1134.86	1109.67
LA22	927	1036.05	1019.98	986.57	971.03	1017.90
LA23	1032	1105.45	1083.27	1073.43	1037.03	1037.65
LA24	935	1047.55	1038.40	1038.63	1037.27	1030.00
Mean	936.07	1028.67	1015.17	990.91	977.87	998.59
Mean gap to LB (%)	–	9.78	8.38	5.73	4.36	6.65

reports $\overline{M}$ over the 30 completed traces; UPPAAL-SMC statistics are analysed separately in Section 6.7.

6.2. Comparative Evaluation on IJSP Benchmarks

6.2.1. Benchmark Performance Comparison

The principal comparison uses 14 benchmark instances: ORB1–ORB5 and LA16–LA24. Nominal schedule quality is measured by the midpoint of the interval makespan. Let the reported midpoint value of method a on instance i be $M_i^{\left(a\right)}$ and let $L{B}_i$ denote the benchmark lower bound. The relative gap for each method–instance pair is defined as

$${Gap}_i^{\left(a\right)}(\%)=100{\displaystyle \frac{M_i^{\left(a\right)}-L{B}_i}{L{B}_i}}.$$

(22)

The mean gap is the arithmetic average of these relative gaps over the 14 instances, with lower values indicating better nominal schedule quality. Descriptive benchmark results are reported here using per-instance midpoint values and mean relative gaps; paired inferential testing across the same 14 instances is reported separately in Section 6.2.2.

The PNS and GA values are taken from the comparison study of Díaz et al. [7], with the original PNS method described by Lei [6]. The fEABC and Fast Elitist Artificial Bee Colony with Local Search (fEABC-LS) results were obtained from our reimplementation of the method in [8] on the same 14 instances. Because this reproduced benchmark set is broader than that reported in the original fEABC study, the corresponding columns should be interpreted as reproduction results rather than transcribed literature values.

RAPO-RL-TAC achieves a mean midpoint makespan of 998.59, improving on PNS and GA by 2.92% and 1.63%, respectively. The reproduced fEABC and fEABC-LS variants achieve lower mean midpoint makespans, by 0.77% and 2.12%, respectively. At the instance level, RAPO-RL-TAC attains the lowest midpoint on LA21 and LA24 while retaining machine-order flexibility for subsequent timed-automata completion. Its mean relative gap to the listed lower bounds is 6.65%.

6.2.2. Statistical Significance Analysis

To assess whether the per-instance differences in Table 2 are systematic across the benchmark set, paired two-sided Wilcoxon signed-rank tests were conducted between RAPO-RL-TAC and each comparison method. The 14 matched benchmark instances constitute the paired observations, with one midpoint value from RAPO-RL-TAC and one corresponding midpoint value from the comparison method for each instance. The 30 TAC traces used to obtain the RAPO-RL-TAC midpoint for an instance are therefore not treated as independent samples in this test. The Wilcoxon procedure provides a non-parametric paired comparison without requiring normally distributed instance-level differences.

Table 3. Paired two-sided Wilcoxon signed-rank tests across the 14 benchmark instances. W/T/L denotes the number of instances on which RAPO-RL-TAC obtains a lower/equal/higher midpoint makespan than the corresponding comparison method.

Comparison	W	p-Value	W/T/L	Sig. ($\mathit{p}<0.05$)	Direction
RAPO-RL-TAC vs. PNS	4.00	<0.001	13/0/1	Yes	RAPO-RL-TAC better
RAPO-RL-TAC vs. GA	13.00	0.011	12/0/2	Yes	RAPO-RL-TAC better
RAPO-RL-TAC vs. fEABC	41.00	0.502	6/0/8	No	No significant difference
RAPO-RL-TAC vs. fEABC-LS	13.00	0.011	3/0/11	Yes	fEABC-LS better

reports the Wilcoxon statistic W, the two-sided p-value, and the win/tie/loss (W/T/L) count from the perspective of RAPO-RL-TAC, where a win denotes a lower midpoint makespan. Statistical significance is assessed at $p<0.05$.

The comparison with PNS is statistically significant ($W=4.00$, $p<0.001$), with RAPO-RL-TAC obtaining a lower midpoint on 13 of the 14 instances. The difference relative to GA is also significant ($W=13.00$, $p=0.011$), with 12 wins and 2 losses. In contrast, the difference between RAPO-RL-TAC and the reproduced fEABC results is not statistically significant ($W=41.00$, $p=0.502$). For fEABC-LS, a significant difference is observed ($W=13.00$, $p=0.011$), but the direction favours fEABC-LS, which obtains a lower midpoint on 11 of the 14 instances. These results refine the descriptive comparison in Table 2: statistical evidence supports the lower midpoint performance of RAPO-RL-TAC relative to PNS and GA, while no significant difference is established relative to fEABC and fEABC-LS remains stronger under the conventional midpoint criterion. Nevertheless, RAPO-RL-TAC achieves lower midpoint makespans than fEABC on 6 of the 14 instances and than fEABC-LS on 3 instances, including the lowest values among all compared methods on LA21 and LA24.

6.3. Random-Seed Robustness Analysis

Training randomness is evaluated separately from the stochasticity of timed-automata execution. For each of the 14 benchmark instances, three independent PPO/GNN training runs were conducted using random seeds 10, 20, and 30. Each run starts from an independently initialised learning process and produces its own learned representations ${\mathbf{P}}^{*}$ and $\mathbf{Q}$. For each training seed, the learned representation is then fixed and evaluated through 30 independent TAC executions using the same procedure defined in Section 6.1. The mean midpoint makespan over these 30 TAC traces is used as the seed-level observation. The three seed-level mean midpoint makespans constitute the independent observations used to summarise training-seed variability for each instance.

Across the three seed-level observations, we report the arithmetic mean, sample standard deviation (Std), and two-sided 95% confidence interval computed using Student’s t distribution with two degrees of freedom. Figure 8 is a horizontal interval plot that summarises these statistics for all 14 benchmark instances. Each row corresponds to one instance: the circular marker denotes the mean midpoint makespan across the three training seeds, the horizontal error bar spans the corresponding 95% confidence interval, and the adjacent annotation reports “Mean[95% CI Lower, 95% CI Upper](Std)”. The vertical dashed line marks the overall mean of the 14 seed-averaged midpoint values, providing a common reference across instances.

The seed-level variability is small relative to the magnitude of the midpoint makespans. Across the 14 instances, the sample standard deviation ranges from 1.53 to 8.51. The coefficient of variation remains below 1% for every instance, with an average of approximately 0.43% and a maximum of approximately 0.90%. The average of the 14 seed-averaged midpoint values is 999.77, which is close to the mean midpoint of 998.59 reported in Table 2; the absolute difference is 1.18, or approximately 0.12%. Under the three tested training seeds, the resulting schedule quality is therefore relatively stable and remains consistent with the performance level observed in the primary benchmark experiment.

6.4. Computational and Structural Efficiency Analysis

To quantify computational cost and the structural reduction produced by the learned partial order, we record training, inference, TAC, and residual-conflict metrics for the same 14 ORB and LA benchmark instances. In the main experimental campaign, multiple instance-specific training jobs are executed concurrently, so their elapsed times are affected by shared computational resources and do not provide a directly comparable per-instance reference. We therefore profile each benchmark instance separately under the same $N_{\mathrm{env}}=10$ within-instance parallel-environment configuration, recording the wall-clock time required for each environment to complete 1000 training episodes. The resulting values are reported as reference measurements of single-instance training cost. Inference time is averaged over repeated greedy-inference measurements, and TAC time is the mean wall-clock time per run over the 30 independent TAC executions for each fixed learned representation.

For structural comparison, let $|E_0|$ denote the number of initial same-machine conflict relations and $|E_T^{\mathrm{open}}|$ the number left unresolved after RAPO-RL inference. For a $10\times 10$ instance, $|E_0|=10\left({\displaystyle \genfrac{}{}{0pt}{}{10}{2}}\right)=450$, whereas for a $15\times 10$ instance, $|E_0|=10\left({\displaystyle \genfrac{}{}{0pt}{}{15}{2}}\right)=1050$. We report the machine-conflict reduction ratio as

$$R_{\mathrm{red}}(\%)=100\left(1-{\displaystyle \frac{|E_T^{\mathrm{open}}|}{|E_0|}}\right).$$

(23)

The resulting computational and structural efficiency metrics are reported in

Table 4. Computational efficiency and structural reduction metrics of RAPO-RL-TAC.

Instance	Train. Time (1000 ep./env., s)	Mean Inf. Time (s)	Mean TAC Time/Run (s)	Initial Conflicts	Unresolved Edges	Reduction (%)
ORB1	3974	1.28	18	450	16	96.44
ORB2	3826	1.32	17	450	18	96.00
ORB3	4083	1.81	17	450	13	97.11
ORB4	3907	2.21	15	450	16	96.44
ORB5	4053	1.44	18	450	19	95.78
LA16	4072	1.27	12	450	12	97.33
LA17	3922	1.29	14	450	16	96.44
LA18	4471	1.26	14	450	13	97.11
LA19	4212	1.38	16	450	12	97.33
LA20	3874	1.29	13	450	18	96.00
LA21	24,384	5.72	15	1050	17	98.38
LA22	24,127	5.85	13	1050	17	98.38
LA23	24,793	5.74	15	1050	19	98.19
LA24	25,109	5.78	15	1050	17	98.38

. The training-time column represents single-instance profiling measurements under the same $N_{\mathrm{env}}=10$ configuration, while inference time and TAC time are obtained from repeated measurements and 30 independent executions, respectively. The table also reports the initial machine-conflict relations, unresolved edges after RAPO-RL inference, and the corresponding reduction ratio.

The single-instance profiling results show a clear increase for the larger $15\times 10$ instances, whereas post-training inference remains below 6 s for all tested instances. Mean TAC execution time remains within 12–18 s per run. During training, the maximum observed peak CPU/GPU memory across instances of the same scale is 1.51/1.79 GB for the $10\times 10$ group and 2.15/8.30 GB for the $15\times 10$ group. These measurements indicate that the dominant computational cost is associated with offline learning, while inference and timed-automata completion incur substantially smaller wall-clock costs under the evaluated environment.

As shown in Figure 9, RAPO-RL leaves only 12–19 unresolved machine-conflict relations for TAC. The reduction ratio ranges from 95.78% to 97.33% for the $10\times 10$ instances, with a mean of 96.60%, and from 98.19% to 98.38% for the $15\times 10$ instances, with a mean of 98.33%. These percentages quantify reduction in machine-conflict relations rather than an exhaustive UPPAAL state-space reduction. Because UPPAAL-SMC evaluates stochastic timed behaviour through statistical sampling rather than exhaustive symbolic enumeration, exact exhaustive state or branch counts are not directly reported; the initial/residual conflict counts, runtime, and memory measurements are therefore used as computational and structural indicators of the residual decision space.

6.5. Ablation Study

Four configurations are compared. Reinforcement-learning-only (RL-only) resolves all machine conflicts through the learned policy and omits timed-automata completion; statistical-model-checking-only (SMC-only) removes the learning stage; Hard-constraint enforces learned dependencies before queue admission; and RAPO-RL-TAC uses the learned tiers only to prioritise currently waiting requests. The RAPO-RL-based variants use the same open-edge budget $cap=3$, so the Hard-constraint comparison isolates how the learned relation is interpreted by the timed-automata model. Figure 10 shows the additional dependency(jid,cur_step) guard used by the Hard-constraint Job template; the dependency must be satisfied before queue admission.

The ablation study selects four instances ABZ9, FT20, LA21, and LA24 to assess the contributions of partial-order learning and timed-automata completion under structurally different job-shop settings. These four instances are drawn from three classical benchmark families, namely ABZ, FT, and LA [49,50,51], and span different job–machine configurations and instance scales, thus forming a representative and heterogeneous subset for mechanism-level evaluation of the proposed framework. ABZ9 originates from the Adams–Balas–Zawack benchmark family and represents a larger-scale benchmark configuration compared with several conventional small-sized instances. FT20 originates from the Fisher–Thompson benchmark family and provides a different job–machine configuration from the other selected instances. LA21 and LA24 are selected from the Lawrence benchmark family and use the $15\times 10$ configuration considered in the main evaluation, providing additional cases from the larger Lawrence instance group. This combination allows the ablation study to examine the contribution of each RAPO-RL-TAC component across distinct scheduling structures, providing a more informative mechanism-level comparison of partial-order learning, timed completion, and hard enforcement.

As shown in Figure 11, RAPO-RL-TAC achieves the lowest midpoint makespan on all four ablation instances, with a mean of 1023.50 compared with 1053.75 for RL-only, 1094.00 for SMC-only, and 1188.00 for Hard-constraint. This corresponds to reductions of 2.87%, 6.44%, and 13.85%, respectively. The largest deterioration occurs under hard enforcement, supporting the use of learned relations as queue-level guidance rather than pre-queue prerequisites. Because the ablation covers four representative instances, these results should be interpreted as mechanism-level evidence rather than a general dominance result.

6.6. Hyperparameter Sensitivity Analysis

A brief sensitivity study is conducted for the temporal-determinacy threshold $\tau$ and the machine-wise open-edge budget $cap$, which are fixed to $\tau =0.05$ and $cap=3$ in the main experiments. The four ablation instances ABZ9, FT20, LA21, and LA24 are used so that the sensitivity analysis remains aligned with the mechanism-level evaluation. For each parameter setting, the same training seed is used and all remaining training settings are held fixed; the resulting learned representation is evaluated using the same 30-run TAC protocol as the main experiments. One parameter is varied at a time, with $\tau \in \{0.03,0.05,0.07\}$ and $cap\in \{1,3,5\}$.

The sensitivity results for $\tau$ and $cap$ are summarised in Figure 12. The figure presents the tested parameter settings and the corresponding midpoint makespan variations relative to the default configuration.

For the threshold study, the four-instance mean midpoint makespans are 1041.00, 1023.50, and 1032.75 for $\tau =0.03$, 0.05, and 0.07, respectively. Thus, $\tau =0.05$ gives the lowest average among the tested values. At the instance level, ABZ9, LA21, and LA24 also attain their lowest midpoint at $\tau =0.05$, whereas FT20 is slightly lower at $\tau =0.07$ (1186) than at the default setting (1193).

The open-edge budget exhibits a clearer pattern over the evaluated settings. The four-instance mean midpoint makespans are 1046.50, 1023.50, and 1080.25 for $cap=1$, 3, and 5, respectively; relative to $cap=3$, these correspond to average increases of approximately 2.25% and 5.54% for $cap=1$ and $cap=5$. The default budget produces the lowest midpoint on all four tested instances. This pattern is consistent with the intended trade-off: a smaller budget can reduce retained temporal flexibility, whereas a larger budget leaves more residual machine-order ambiguity for timed completion.

6.7. Statistical Model Checking Assessment

The SMC experiment complements the midpoint benchmark by characterising timed executions after $\mathbf{Q}$ is transferred to TAC. UPPAAL-SMC reports completion-time distributions, sample statistics, confidence information, and time-bounded completion probabilities, thereby providing temporal information that is not represented by the scalar midpoint criterion. Time-bounded completion probability is conceptually aligned with makespan service-level measures used in uncertain job-shop research [46].

Two representative cases, ORB2 and FT10, are evaluated using a common time bound of $T=1300$.

As shown in Figure 13, all sampled executions in both cases finish within $T=1300$, yielding ${\widehat{P}}_T=1.000$. ORB2 exhibits a markedly wider observed completion-time range than FT10, indicating greater temporal variability under the sampled UPPAAL-SMC execution semantics.

6.8. Discussion

The benchmark and statistical comparisons show that RAPO-RL-TAC maintains competitive nominal schedule quality across the evaluated IJSP instances while retaining timing-sensitive machine-order relations for subsequent completion. Its mean midpoint makespan is 998.59, compared with 1028.67 for PNS and 1015.17 for GA, corresponding to mean improvements of 2.92% and 1.63%, respectively. The paired Wilcoxon tests further show that these differences are statistically significant: RAPO-RL-TAC obtains lower midpoint makespans than PNS on 13 of the 14 matched instances ($W=4.00$, $p<0.001$) and than GA on 12 instances ($W=13.00$, $p=0.011$). The reproduced fEABC obtains a lower mean midpoint of 990.91, but the difference from RAPO-RL-TAC is only 0.77% and is not statistically significant ($W=41.00$, $p=0.502$). The reproduced fEABC-LS achieves the lower overall mean midpoint of 977.87 and is statistically favoured in the paired comparison ($W=13.00$, $p=0.011$), with a lower midpoint on 11 of the 14 instances. At the same time, RAPO-RL-TAC obtains lower midpoint makespans than fEABC on 6 instances and than fEABC-LS on 3 instances, and achieves the lowest midpoint among all compared methods on LA21 and LA24. Taken together, these results show that RAPO-RL-TAC remains competitive under the conventional midpoint objective while preserving the partial-order flexibility required by its timed-completion mechanism.

The random-seed analysis indicates that the observed schedule quality is relatively stable across the three independently trained PPO/GNN seeds. Across the 14 benchmark instances, the sample standard deviations range from 1.53 to 8.51, while the coefficient of variation remains below 1% for every instance, with an average of approximately 0.43% and a maximum of approximately 0.90%. Moreover, the overall mean of the seed-averaged midpoint values is 999.77, only approximately 0.12% different from the mean value of 998.59 in the primary benchmark experiment. These results indicate limited performance variation under the three tested training seeds.

The computational and structural measurements further clarify where the workload of the framework is concentrated and how much scheduling ambiguity is retained for TAC. In the single-instance reference profiles, training time ranges from 3826 to 4471 s for the $10\times 10$ instances and from 24,127 to 25,109 s for the $15\times 10$ instances. Post-training inference remains below 6 s for all evaluated instances, and mean TAC execution time ranges from 12 to 18 s per run. The observed peak CPU/GPU memory increases from 1.51/1.79 GB for the $10\times 10$ group to 2.15/8.30 GB for the $15\times 10$ group, indicating that the measured computational burden is concentrated primarily in offline training. Structurally, only 12–19 same-machine conflict relations remain unresolved after RAPO-RL inference. The mean machine-conflict reduction reaches 96.60% for the $10\times 10$ instances and 98.33% for the $15\times 10$ instances. These results show that RAPO-RL resolves the large majority of same-machine conflicts before timed execution while retaining a small subset for state-dependent completion. The reported percentages quantify reduction in machine-conflict relations and do not represent exhaustive UPPAAL state-space reduction.

The ablation study provides direct mechanism-level evidence for the complementary roles of partial-order learning and timed completion. Across ABZ9, FT20, LA21, and LA24, RAPO-RL-TAC achieves a mean midpoint makespan of 1023.50, compared with 1053.75 for RL-only, 1094.00 for SMC-only, and 1188.00 for Hard-constraint, corresponding to reductions of 2.87%, 6.44%, and 13.85%, respectively. RAPO-RL-TAC also obtains the lowest midpoint on each of the four evaluated instances. The deterioration observed when either the learned partial-order stage or the timed-completion stage is removed supports the complementary contribution of the two stages. The substantially larger deterioration under Hard-constraint further supports using the learned precedence information as queue-level guidance, allowing TAC to retain state-dependent flexibility during execution. The sensitivity results are consistent with this interpretation. For $\tau =0.03$, 0.05, and 0.07, the four-instance mean midpoint makespans are 1041.00, 1023.50, and 1032.75, respectively, so the default $\tau =0.05$ provides the lowest average among the tested thresholds, although FT20 is slightly better at $\tau =0.07$. For $cap=1$, 3, and 5, the corresponding means are 1046.50, 1023.50, and 1080.25, and $cap=3$ yields the lowest midpoint on all four tested instances. Within the evaluated ranges, these patterns indicate that the adopted settings provide a reasonable balance between resolving sufficiently determinate conflicts before execution and retaining a controlled amount of temporal flexibility for TAC.

The UPPAAL-SMC analysis provides a complementary view of schedule behaviour under interval-valued processing times. For both ORB2 and FT10, all sampled executions complete within the common bound $T=1300$, giving ${\widehat{P}}_T=1.000$. However, their observed completion-time distributions differ substantially: ORB2 spans approximately 942.91–1175.33, whereas FT10 spans approximately 938.20–1028.73. The markedly wider range for ORB2 indicates greater temporal variability under the sampled execution semantics. These results illustrate that statistical model checking can reveal distributional and time-bounded execution characteristics that are not represented by the scalar fixed-sequence midpoint used in the benchmark comparison.

Several limitations of the present study should be acknowledged. First, RAPO-RL is currently trained separately for each benchmark instance, and cross-instance generalisation has therefore not been evaluated. Second, information currently flows only from RAPO-RL to TAC, and the outcomes of formal temporal analysis are not used to update the learned policy. Third, owing to time and computational-resource constraints, the random-seed analysis is based on three independent PPO/GNN training seeds, and the hyperparameter sensitivity study considers three values for each parameter on four ablation instances. These evaluations provide evidence on training-seed variability and local parameter behaviour within the tested settings, while broader validation across additional training runs and parameter configurations remains necessary.

7. Conclusions

This study established RAPO-RL-TAC, a reinforcement-learning-guided formal framework for the Interval Job Shop Problem (IJSP), demonstrating that selectively partitioning machine-order decisions between RAPO-RL and TAC effectively balances schedule quality and temporal flexibility under interval uncertainty. By resolving most machine conflicts prior to execution, this approach substantially reduces the residual decision space while retaining critical uncertainty-sensitive relations. Comparative experiments confirm that this selective strategy preserves essential scheduling flexibility when optimal sequences depend on actual temporal realizations, while maintaining competitive nominal schedule quality. Furthermore, the ablation results show that integrating RAPO-RL and TAC effectively reduces the makespan of IJSP solutions. Statistical model checking complements the optimisation results by revealing completion-time variability across sampled timed executions, providing a broader view of schedule behaviour under interval uncertainty. In conclusion, RAPO-RL-TAC provides a robust methodology for mitigating decision complexity, preserving sequence flexibility, and rigorously assessing temporal performance in the IJSP.

Future work will focus on extending the framework to larger and more diverse IJSP instances, evaluating cross-instance generalisation, conducting broader random-seed and parameter studies, and developing bidirectional interaction between RAPO-RL and TAC so that outcomes from formal temporal analysis can enhance learning-based scheduling decisions.