CReSCENT for Long-Term Value-Driven Scheduling in Multi-Layer Industrial Networks with Milestone-Triggered Rewards

Abstract

This paper studies long-term value-driven scheduling in multi-layer industrial networks where useful external reward is released mainly at milestone transitions. The delayed feedback causes supervision degeneracy before milestones, because many trajectory prefixes receive the same external return even when their downstream potential is different. We formulate the setting as a time-delayed multi-industrial-chain Markov decision process and present CReSCENT as a milestone-aware structural exploration framework rather than a new reinforcement learning principle. CReSCENT combines macro structural feature construction, contrastive representation learning, milestone-weighted online clustering, and cross-layer credit allocation. It moves intrinsic learning from raw observations to milestone-relevant structural states, then distributes the resulting signal to layers according to their contribution to cross-layer progress. The revised evaluation gives the simulator transition rules, a formal utility metric, an external-reward-only baseline, confidence intervals, and statistical tests. Experiments show that CReSCENT improves utility across layer, task-load, worker-count, and episode-length perturbations against ETA-PSI, EMU, MIMEx, and the external-only baseline. Sensitivity studies further show that the clustering radius and credit-allocation weights have stable operating ranges.

1. Introduction

Multi-layer industrial networks place scheduling decisions in a setting where physical processing, transport, buffer evolution, and inter-layer dependencies unfold at different speeds. A dispatching action in one layer may look harmless at the moment it is issued, yet its real value can be revealed only after upstream queues have been cleared, downstream capacity has become available, and a group of tasks has crossed a structural milestone. This delayed reward pattern differs from the step-continuous feedback usually assumed in industrial cyber-physical scheduling studies [1,2]. It makes the scheduler learn from sparse and uneven signals while the underlying state still changes at every step.

This paper studies long-term value-driven scheduling under reward delay in multi-layer industrial networks. The core observation is that a delayed reward is not merely a low-frequency version of an ordinary reward. Before a milestone is triggered, many trajectory prefixes receive nearly indistinguishable external feedback even when their long-term potentials are very different. A policy trained only on external returns may therefore over-reinforce a path that produces an early but incomplete improvement, while missing a path that requires longer cross-layer preparation and later yields a larger structural gain. This phenomenon is particularly damaging in networks where each layer is controlled by a local agent, because the final milestone is the result of multiple agents acting across a chain of queues and resources.

Existing reinforcement learning schedulers help with model dependence and dynamic arrivals, but they still rely on reward signals that can be attributed to actions over a manageable horizon [3,4]. Multi-agent value factorisation and actor-critic methods improve coordination under cooperative rewards [5,6], yet they do not by themselves create a learning signal for prefixes that are structurally promising but externally silent. Intrinsic motivation methods add exploration bonuses from prediction error, random network distillation, or count-based novelty [7,8,9,10]. Their novelty is usually defined on instantaneous observations or local latent states. In a multi-layer industrial network, this is too weak, because the relevant novelty lies in the cross-layer structural progress that leads to milestone activation.

We propose CReSCENT, which stands for Contrastive Representation and State Clustering for Exploration via Novelty-based Training. The contribution is a milestone-aware structural exploration framework for delayed industrial scheduling rather than a fundamentally new reinforcement learning principle. CReSCENT lifts the intrinsic-learning signal from raw observations to macro structural states. It first constructs cross-layer features that describe queues, resource loads, blocking relations, and milestone progress. It then learns a contrastive latent representation in which structurally similar states are close and structurally different states are separated. An online clustering module converts the latent stream into pseudo counts and weights the count bonus by cluster-level milestone usefulness, so structurally new but unproductive regions do not receive the same long-term emphasis as milestone-relevant regions. Finally, a cross-layer credit allocation mechanism distributes the global novelty reward to layer-specific updates according to each layer’s contribution to milestone progress.

The contributions are as follows.

• A formal TD-MIC-MDP model for milestone-triggered multi-layer industrial scheduling, with a proposition showing why external reward alone degenerates before the milestone is reached.
• A structural exploration framework that combines macro feature construction, contrastive representation learning, milestone-weighted online pseudo-count clustering, and interpretable cross-layer credit allocation.
• A reproducible simulator specification that gives the transition graph, arrival rules, milestone-progress rules, external reward, utility metric, random seeds, and implementation settings used in the experiments.
• Experiments showing that CReSCENT improves utility and reward against ETA-PSI, EMU, MIMEx, and an external-reward-only baseline under layer, task, worker, and episode-length perturbations.
• Statistical tests, confidence intervals, clustering-threshold sensitivity, credit-weight sensitivity, and revised latent-space diagnostics based on t-SNE, UMAP, and PaCMAP.

2. Related Work

Industrial cyber-physical scheduling has moved from isolated shop-floor dispatching toward networked production in which sensors, platforms, and distributed resources are coupled across layers. Industry 4.0 surveys emphasise connectivity, cyber-physical integration, and open optimisation issues [1,2]. Multi-factory scheduling reviews further show that planning decisions must balance local production, transport, and network-level coordination [11]. Deep reinforcement learning has been used for global production scheduling because it can learn dispatching policies from interaction rather than relying on a fixed analytical model [12]. Adjacent resource-orchestration studies in edge service chains, distributed data centers, and fluctuating data streams further illustrate the need to coordinate online tasks across coupled computing or service layers [13,14,15]. These studies motivate learning-based scheduling, but most of them assume that feedback remains dense enough to guide every update.

The present work is closer to delayed-reward industrial scheduling than to general production planning. For that reason, the revised related work focuses on three directly supporting lines: reinforcement learning under delayed or sparse reward, multi-agent coordination with shared outcomes, and representation-based intrinsic motivation. This narrower scope avoids using weakly related scheduling papers merely as background and makes the role of each cited work explicit.

Reinforcement learning provides the decision-theoretic basis for long-horizon optimisation [3,16]. Temporal-difference learning explains how value estimates can be refined from delayed observations [17], and reward shaping theory clarifies when auxiliary reward can preserve the optimal policy [18]. Modern policy-gradient methods such as GAE and PPO improve stability in high-dimensional control [4,19], while maximum entropy learning encourages broader exploration [20]. Survey work on deep reinforcement learning summarises these algorithmic families and their tradeoffs [21]. CReSCENT builds on this foundation but focuses on the missing supervision before milestone rewards become visible.

Multi-agent reinforcement learning addresses decentralised decisions with shared or coupled outcomes. Early surveys describe the non-stationarity and coordination challenges of multi-agent learning [22,23]. Recent surveys extend this discussion to deep policies and industrial-scale applications [24,25]. Actor-critic methods with centralised training and decentralised execution improve cooperative control [5,26,27]. Value-decomposition methods learn a joint value through agent-level terms [6,28,29]. Benchmarking and MAPPO studies show that implementation details and stable policy updates can dominate performance in cooperative games [30,31]. These methods help coordination, but they do not define a milestone-aware novelty signal for structurally silent trajectory prefixes.

Intrinsic motivation is the closest line of work to CReSCENT. Curiosity, random network distillation, and count-based novelty create internal learning signals when external rewards are sparse [7,8,9,10]. Noisy networks, episodic return-and-explore mechanisms, and Agent57-style exploration improve state-space coverage [32,33,34]. More recent work studies when to explore in single-agent and multi-agent settings [35,36]. Cooperative intrinsic reward methods further use state entropy, episodic memory, masked input modelling, or variational exploration to guide multi-agent behaviour [37,38,39,40]. CReSCENT differs by defining novelty in a macro structural representation tied to milestone reachability.

Representation learning gives a second foundation for the proposed structural state space. Contrastive predictive coding and SimCLR show that discriminative latent objectives can organise observations by predictive or semantic similarity [41,42]. Variational autoencoders and disentanglement objectives provide latent variables with controllable capacity [43,44]. Recurrent encoders model ordered evidence across layers or time [45]. Visualisation methods such as t-SNE, UMAP, and PaCMAP help inspect whether the latent representation separates relevant operating states [46,47,48]. Belief-state planning, Bayesian reinforcement learning, safe reinforcement learning, and hierarchical decomposition provide related views of uncertainty, partial observability, risk, and temporal abstraction [49,50,51,52]. CReSCENT uses these ideas in a narrower way, with representation quality judged by structural progress toward delayed milestones.

Table 1. Related work summary for long-term value-driven scheduling.

Area	Representative Work	Main Modelling Focus	Relation to CReSCENT
Industrial network scheduling	[1,2,11,12]	Connected production, cyber-physical integration, multi-factory planning, and learned dispatching	Provides the application setting, while CReSCENT targets sparse milestone feedback inside such networks
RL foundations	[3,16,17,18]	Sequential decision making, temporal-difference credit, Markov processes, and shaping invariance	Supplies the objective basis, while the proposed method adds structural supervision before delayed rewards arrive
Policy optimisation	[4,19,20,21]	Advantage estimation, clipped updates, entropy regularisation, and deep RL design choices	Supplies stable optimisers that can receive the CReSCENT intrinsic signal
MARL surveys	[22,23,24,25]	Coordination, non-stationarity, scalability, and application challenges	Clarifies why delayed milestones become harder when each layer is controlled by an agent
Actor-critic coordination	[5,26,27]	Centralised training, counterfactual credit, and trust-region control	Improves cooperative optimisation but does not define structural novelty under silent prefixes
Value factorisation	[6,28,29,30,31]	Joint value decomposition, monotonic mixing, transformed factorisation, and benchmark practice	Gives cooperative baselines, while CReSCENT targets the reward-observation gap before milestones
Intrinsic motivation	[7,8,9,10,32]	Curiosity, prediction error, density novelty, pseudo counts, and parameter noise	Inspires internal reward, while CReSCENT measures novelty in milestone-relevant macro states
Exploration timing	[33,34,35,36]	Episodic memory, return-first search, and learned switching between exploration modes	Helps reason about when to explore, while CReSCENT allocates novelty by cross-layer structural progress
Recent cooperative exploration	[37,38,39,40]	Entropy coverage, episodic memory, masked modelling, and variational exploration	Provides direct baselines for intrinsic multi-agent learning
Representation learning	[41,42,43,44,45]	Contrastive objectives, latent variables, disentanglement, and recurrent encoding	Supports the structural encoder used to cluster macro progress states
Belief and structure	[46,47,48,49,50,51,52]	Latent inspection, partial observability, Bayesian uncertainty, safe learning, and hierarchy	Motivates analysing representation geometry and maintaining interpretable structural progress

summarises the relationship between the relevant literature and the proposed framework.

3. Problem Formulation

3.1. Milestone-Triggered Reward

Consider a multi-layer industrial network with layers indexed by $\ell \in \{0,\dots ,M-1\}$. At time t, layer ℓ observes $o_t^{\ell }$ and selects a dispatching action $a_t^{\ell }$. The joint action is $a_t=(a_t^0,\dots ,a_t^{M-1})$. The environment state $s_t$ contains task queues, worker availability, processing progress, inter-layer buffers, and global structural indicators. A transition $s_t\to s_{t+1}$ may trigger a milestone when a cross-layer condition becomes satisfied. Let

$${\mathbb{I}}_t=\mathbb{I}\left(s_t\to s_{t+1}\mathrm{triggers}\mathrm{a}\mathrm{milestone}\right)\in \{0,1\}.$$

(1)

The external reward is

$$r_t^{ext}={\mathbb{I}}_t{r}^{mil}(s_t,a_t,s_{t+1})+r^{aux}(s_t,a_t,s_{t+1}),$$

(2)

where $r^{mil}$ is a structural reward released at milestone transitions, and $r^{aux}$ is a weak auxiliary term used only to prevent pathological idling. In the pure delayed-reward regime $r^{aux}$ may be zero. Most transitions then supply no meaningful external distinction between candidate prefixes.

3.2. TD-MIC-MDP

We define the time-delayed multi-industrial-chain Markov decision process as

$$\begin{array}{cc}{\mathcal{M}}^{td}=& {\left(\right\{{\mathcal{O}}^{\ell }\}}_{\ell =0}^{M-1},\mathcal{S},{\left\{{\mathcal{A}}^{\ell }\right\}}_{\ell =0}^{M-1},\\ \hfill & P,R^{td},\alpha ,\beta ,\gamma ).\hfill \end{array}$$

(3)

The observation space ${\mathcal{O}}^{\ell }$ describes the local information of layer ℓ. The state space $\mathcal{S}$ contains the full network state. The action space ${\mathcal{A}}^{\ell }$ contains the resource assignment and sequencing choices of layer ℓ. The transition kernel $P(·\mid s_t,a_t)$ includes task arrival, within-layer processing, and migration to the next layer. The reward map $R^{td}$ follows (2). The parameters $\alpha$ and $\beta$ weight cost and utility terms inside the simulator, while $\gamma$ is the discount factor.

Each layer uses a decentralised policy ${\pi }^{\ell }(a^{\ell }\mid o^{\ell })$ and the joint policy is $\pi ={\left\{{\pi }^{\ell }\right\}}_{\ell =0}^{M-1}$. The external objective remains

$$\begin{array}{c}\hfill {\pi }^{*}=arg\underset{\pi }{max}{\mathbb{E}}_{\pi }\left[{\displaystyle \sum _{t=0}^{\infty }}{\gamma }^t{R}^{td}(s_t,a_t)\right].\end{array}$$

(4)

The difficulty is that the objective is observed through sparse structural events. The agents must learn a long-horizon preference over prefixes that may not differ in immediate reward.

3.3. Evaluation Utility

The utility used in the experiments is a system-level episode score rather than an undefined shorthand. For episode e, let $N_e^{mil}$ be the number of triggered milestones, $N_e^{out}$ be the completed task workload, $C_e^{proc}$ be processing and switching cost, $C_e^{wait}$ be waiting cost, and $B_e^{end}$ be terminal backlog. The reported utility is

$$U_e=w_m{N}_e^{mil}+w_y{N}_e^{out}-w_p{C}_e^{proc}-w_w{C}_e^{wait}-w_b{B}_e^{end}.$$

(5)

The default weights are $w_m=1200$, $w_y=8$, $w_p=0.10$, $w_w=0.05$, and $w_b=1$. The milestone term is intentionally the largest term because the environment is designed to evaluate long-horizon structural completion. The throughput term rewards productive completion between milestones. The three penalty terms prevent a policy from obtaining high utility by creating excessive processing cost, long waiting queues, or unreleased backlog. All methods are evaluated with the same weights, and these weights are not used differently by CReSCENT during training.

3.4. Supervision Degeneracy Before Milestones

Proposition 1

(Supervision degeneracy before milestone activation). Assume that

$$r_t^{ext}={\mathbb{I}}_t{r}^{mil}(s_t,a_t,s_{t+1})+r_t^{aux}.$$

If ${\mathbb{I}}_t=0$ over most intermediate transitions and $r_t^{aux}$ is zero, constant, or weakly penalising, then two trajectory prefixes that have not yet triggered a milestone can obtain identical or nearly identical cumulative external reward even when their long-term reachability to high-value milestones is different.

Proof. 

For a prefix ${\tau }_{0:T-1}$, the cumulative external return is

$$G_{0:T-1}^{ext}={\displaystyle \sum _{t=0}^{T-1}}{\gamma }^t{r}_t^{ext}.$$

If no milestone is triggered, then the main term ${\mathbb{I}}_t{r}^{mil}$ vanishes for every step. The return is therefore determined only by $r_t^{aux}$. Since the auxiliary term is not designed to encode cross-layer structural progress, two prefixes with different downstream milestone reachability can have the same or nearly the same return. A learner that updates solely from this return cannot form a stable preference between them before later milestone evidence arrives.    □

Figure 1 illustrates the distinction. In a short-delay setting, feedback is propagated step by step and early actions can be ranked quickly. In a milestone-triggered setting, both candidate prefixes may remain externally silent until lower layers complete the cross-layer condition. The early policy gradient therefore has weak guidance exactly when exploration direction is most important.

4. CReSCENT Method

4.1. Overview

CReSCENT treats delayed reward as a structural representation problem. Instead of assigning novelty to raw observations, it assigns novelty to latent states that describe cross-layer progress toward milestone activation. The method contains four modules. Macro structural feature construction maps raw observations to $x_t$. A contrastive encoder maps $x_t$ to a latent state $z_t$. Milestone-weighted online clustering turns the latent stream into pseudo counts and produces a global intrinsic learning signal $r_t^{int}$. Cross-layer credit allocation distributes this signal to every layer, giving $r_t^{int,\ell }$. Each layer then trains with

$$r_t^{\ell }=r_t^{ext,\ell }+\eta r_t^{int,\ell },$$

(6)

where $\eta$ controls the strength of the intrinsic component. The policy optimiser can be HAPPO or another cooperative multi-agent policy-gradient method [27]. The contribution of CReSCENT is the construction and routing of the learning signal, not a change in the base optimiser.

4.2. Macro Structural Feature Construction

Let $q_t^{\ell }$ denote the queue state of layer ℓ, $b_t^{\ell }$ its local buffer occupancy, $u_t^{\ell }$ its worker utilisation, $p_t^{\ell }$ its aggregate processing progress, and $g_t^{\ell }$ a local milestone-proximity indicator. CReSCENT constructs a macro feature vector

$$x_t=\mathsf{\Phi }\left(s_t\right)=\left[{\psi }_0\left(s_t\right),{\psi }_1\left(s_t\right),\dots ,{\psi }_{K_s-1}\left(s_t\right)\right],$$

(7)

where $\mathsf{\Phi }$ concatenates layer-level summaries, cross-layer differences, and global progress statistics. A typical feature block contains queue length, mean remaining processing time, available worker ratio, upstream-to-downstream backlog difference, inter-layer blocking count, and the fraction of tasks that have reached each milestone stage.

The point of this map is not dimensionality reduction alone. It removes observation details that are irrelevant to delayed value while retaining the structural variables that determine whether a future milestone is reachable. For example, two raw states can have different task identifiers yet represent the same macro progress if their queues, bottleneck locations, and milestone proximity are equivalent. A novelty signal based on raw identity would over-count them as different. A novelty signal based on $\mathsf{\Phi }\left(s_t\right)$ treats them as the same region and decays appropriately. Figure 2 shows the overall CReSCENT framework.

4.3. Contrastive Structural Encoder

The contrastive encoder $f_{\theta }$ maps $x_t$ to

$$z_t={\displaystyle \frac{f_{\theta }\left(x_t\right)}{\parallel f_{\theta }\left(x_t\right){\parallel }_2}}.$$

(8)

Positive pairs are produced from two augmentations of the same macro state or from temporally adjacent states whose structural indicators remain within a small tolerance. Negative pairs are sampled from states with different milestone stages, different bottleneck layers, or sufficiently separated queue profiles. For a batch of B positive pairs, the NT-Xent loss is

$${\mathcal{L}}_{con}=-{\displaystyle \frac{1}{2B}}{\displaystyle \sum _{i=1}^{2B}}log{\displaystyle \frac{exp(\mathrm{sim}(z_i,z_{p\left(i\right)})/\tau )}{{\sum }_{j\ne i}exp(\mathrm{sim}(z_i,z_j)/\tau )}}.$$

(9)

Here $p\left(i\right)$ indexes the positive counterpart of sample i, $\tau$ is the temperature, and $\mathrm{sim}$ is cosine similarity. This objective follows the broad idea of contrastive representation learning [41,42], but its sampling policy is tied to industrial structure. The learned space is expected to align with milestone reachability rather than visual or sensory similarity.

4.4. Milestone-Weighted Online Clustering and Pseudo-Count Novelty

After encoding, CReSCENT maintains a set of online cluster centres ${\mathcal{C}}_t=\{c_1,\dots ,c_K\}$ in the latent space. A new latent point $z_t$ is assigned to the nearest centre when the Euclidean distance in the unit-normalised latent space is below the clustering threshold $\rho$. A small $\rho$ creates many fine clusters and can fragment one meaningful structural stage. A large $\rho$ merges different bottleneck states and weakens the novelty signal. Otherwise, a new centre is created if the cluster budget allows it, or the weakest centre is replaced according to age and visit statistics. Let $k_t$ be the assigned cluster. The visit count is updated as

$$N_{k_t}\leftarrow N_{k_t}+1.$$

(10)

To avoid rewarding structurally new but milestone-irrelevant states indefinitely, CReSCENT also maintains a cluster-importance estimate. Let ${\overline{g}}_k$, ${\overline{y}}_k$, and ${\overline{h}}_k$ denote exponential moving averages of milestone proximity, completed workload, and backlog pressure observed when cluster k is visited. The usefulness of cluster k is

$$I_k=\sigma \left(v_m{\overline{g}}_k+v_y{\overline{y}}_k-v_h{\overline{h}}_k\right),$$

(11)

where $\sigma (·)$ is the logistic function and $v_m$, $v_y$, and $v_h$ are non-negative scaling coefficients. The global intrinsic signal becomes

$$r_t^{int}={\displaystyle \frac{{\eta }_0{I}_{k_t}}{\sqrt{N_{k_t}+ϵ}}},$$

(12)

with ${\eta }_0>0$ and a small $ϵ$. The signal is high when the policy reaches a new and useful structural region, but it is reduced for clusters that are novel only in geometry and do not move the system toward milestone completion. This behaviour is important in delayed-reward scheduling. Early training needs broad structural coverage so that milestones can be triggered often enough to be learned. Later training needs the novelty signal to weaken so that the policy does not keep wandering after useful milestone paths have been identified.

For an episode of length T, the cumulative intrinsic reward of cluster k is bounded by

$${\displaystyle \sum _{i=1}^{N_k}}{\displaystyle \frac{{\eta }_0{I}_k}{\sqrt{i}}}\le 2{\eta }_0{I}_k\sqrt{N_k}.$$

(13)

Summing across clusters gives a sublinear bound in repeated visits. Thus the intrinsic reward cannot grow linearly by cycling inside one cluster. It must either discover additional structural regions or gradually yield control to external milestone reward.

4.5. Cross-Layer Credit Allocation

A global novelty signal is not enough when the decision makers are layered. If every layer receives the same intrinsic reward, layers that did little to create milestone progress may be updated as strongly as layers that removed the active bottleneck. CReSCENT computes a contribution weight for every layer. Let $\mathsf{\Delta }{m}_t^{\ell }$ denote the change in a layer-level milestone-proximity statistic, $\mathsf{\Delta }{b}_t^{\ell }$ denote the reduction in blocking or backlog imbalance attributable to layer ℓ, and $\mathsf{\Delta }{u}_t^{\ell }$ denote the change in useful worker utilisation. These three terms are normalised to $[-1,1]$ by the corresponding reference range in the simulator. The raw contribution is

$${\omega }_t^{\ell }=\mathrm{ReLU}(a_m\mathsf{\Delta }{m}_t^{\ell }+a_b\mathsf{\Delta }{b}_t^{\ell }+a_u\mathsf{\Delta }{u}_t^{\ell }),$$

(14)

where $a_m,a_b,a_u$ are non-negative coefficients. The default setting uses $a_m=0.50$, $a_b=0.30$, and $a_u=0.20$. The largest weight is assigned to milestone proximity because it is the direct delayed objective. Bottleneck relief is second because queue blockage is the usual cause of milestone delay. Useful utilisation is included with a smaller weight so that the method rewards productive capacity use without treating busy but unhelpful processing as progress. The normalised contribution is

$${\chi }_t^{\ell }={\displaystyle \frac{{\omega }_t^{\ell }+ϵ}{{\sum }_{j=0}^{M-1}({\omega }_t^j+ϵ)}}.$$

(15)

Layer ℓ receives

$$r_t^{int,\ell }={\chi }_t^{\ell }{r}_t^{int}.$$

(16)

This allocation turns a global novelty discovery into layer-specific training pressure. When a downstream layer releases a bottleneck, it receives a larger intrinsic share. When an upstream layer merely adds jobs into an already blocked queue, its contribution is small. The mechanism reduces cross-layer interference because it avoids updating every layer as if every structural novelty were equally caused by every local action.

Table 2. Illustrative cross-layer credit allocation under three structural transitions.

Transition Type	Layer 1 Share	Layer 2 Share	Layer 3 Share
Early queue relief	0.56	0.31	0.13
Midstream bottleneck removal	0.18	0.64	0.18
Milestone release	0.12	0.24	0.64

gives a concrete example. In the first transition, layer 1 clears an upstream queue and obtains most of the intrinsic share. In the second transition, layer 2 removes the dominant bottleneck and receives the largest share. In the third transition, layer 3 triggers the final milestone release and receives most of the signal. The example shows that the mechanism is not a uniform broadcast bonus. It routes the same global novelty event to the layer that most plausibly created the structural progress.

4.6. Training Process

The encoder and clusterer are updated online. This is necessary because the state distribution changes as the policy improves. A fixed representation would either over-emphasise early random exploration or fail to separate the later fine-grained regions where milestone progress depends on subtle cross-layer conditions. The online update keeps the novelty estimate aligned with the current distribution. Algorithm 1 summarises the CReSCENT training process.

Algorithm 1 CReSCENT training process

Require:  Environment, policy set $\left\{{\pi }^{\ell }\right\}$, encoder $f_{\theta }$, cluster budget, intrinsic coefficient $\eta$   1: for each training iteration do   2:       Collect trajectories with the current multi-layer policies   3:       for each transition t do   4:             Construct macro feature $x_t=\mathsf{\Phi }\left(s_t\right)$   5:             Encode $z_t=f_{\theta }\left(x_t\right)/{\parallel f_{\theta }\left(x_t\right)\parallel }_2$   6:             Assign $z_t$ to an online cluster and update its pseudo count   7:             Compute $r_t^{int}$ from the pseudo count   8:             Compute layer weights ${\chi }_t^{\ell }$ and rewards $r_t^{\ell }=r_t^{ext,\ell }+\eta {\chi }_t^{\ell }{r}_t^{int}$   9:       end for 10:       Update $f_{\theta }$ with the contrastive loss 11:       Update the multi-agent policies with the layer-specific rewards 12: end for

4.7. Design Implications

CReSCENT changes the exploration problem in three ways. First, it makes exploration structural. Novelty is not a surprise in a single observation, but a new configuration of queues, blockers, and milestone proximity. Second, it makes exploration self-limiting. Pseudo counts decay with repeated visits and therefore reduce the risk that intrinsic reward permanently dominates the external objective. Third, it makes exploration layer-aware. Cross-layer credit allocation prevents a structural discovery from being broadcast uniformly to layers that did not contribute equally.

5. Experiments

5.1. Setup

The simulator is a multi-layer industrial scheduling environment with serial processing layers, stochastic task arrivals, finite worker pools, inter-layer buffers, and milestone-triggered external rewards.

Table 3. Simulator transition specification for reproducibility.

Element	Specification
Transition graph	Serial directed chain $L_1\to L_2\to \dots \to L_M$ with finite buffers between adjacent layers
Task entry	New tasks arrive at $L_1$ according to a Poisson process with rate $\lambda$ and are rejected only when the entry queue exceeds the capacity limit
Worker rule	Each idle worker selects at most one task from its local queue and processes it for one time step according to the layer-specific processing-time counter
Inter-layer transfer	A completed task advances to the next layer if the downstream buffer is not full; otherwise it remains blocked and contributes to $B_e^{end}$ and blocking statistics
Milestone rule	A milestone is triggered when a target number of tasks reaches the prescribed layer stage or when the active cross-layer backlog target is reduced below its threshold
External reward	Milestone reward is released only at triggered milestones; weak auxiliary penalties discourage idling, excessive waiting, and terminal backlog
Training budget	Ten independent seeds are used for terminal evaluation; each method uses the same backbone optimiser, episode budget, observation access, and action constraints

gives the executable transition specification used in the experiments. The default setting uses three layers, eight workers per layer, Poisson arrivals with rate $\lambda =2.5$, buffer capacity 20 per inter-layer edge, and an episode length of 100 steps. Processing times are sampled from a bounded discrete distribution with layer-dependent means. A task moves from layer ℓ to layer $\ell +1$ only when processing at layer ℓ is complete and the next buffer has capacity. A milestone is triggered when a task batch reaches a predefined completion stage or when a cross-layer backlog target is cleared. The external reward follows Equation (2), while the reported utility follows Equation (5).

All methods use the same outer cooperative policy optimisation backbone. The compared methods are ETA-PSI, EMU, MIMEx, and an external-reward-only baseline. The external-only baseline removes intrinsic reward and trains the same policy backbone with $r_t^{\ell }=r_t^{ext,\ell }$. ETA-PSI represents maximum state entropy exploration with predecessor and successor representations [37]. EMU uses episodic memory to promote desirable cooperative transitions [38]. MIMEx generates intrinsic rewards from masked input modelling [39]. These baselines cover no-intrinsic, entropy-oriented, memory-oriented, and self-supervised intrinsic learning mechanisms under identical simulator access.

5.2. Learning Curves

Figure 3 shows the default learning curves from the simulator. CReSCENT enters the rising phase earlier than the baselines and reaches a higher utility plateau. The cost curve remains controlled, which indicates that the method does not gain reward by simply increasing process activity. Instead, the structural novelty signal increases the frequency of useful milestone triggers, and the cross-layer allocation keeps the added learning pressure aligned with useful layers.

5.3. Robustness Under Structural Perturbations

Table 4. Performance under environment perturbations. Values are mean and standard deviation.

Condition	Metric	CReSCENT	EMU	ETA-PSI	MIMEx	Ext-Only
standard	Reward	$3465.85\pm 28.57$	$3320.02\pm 87.60$	$3276.01\pm 118.45$	$2566.53\pm 108.11$	$2836.74\pm 162.30$
standard	Cost	$9533.58\pm 65.61$	$9486.34\pm 61.05$	$9549.14\pm 100.84$	$9518.94\pm 638.19$	$9440.26\pm 241.18$
standard	Utility	12,334.89$\pm \mathbf{79.84}$	12,003.16 $\pm 143.73$	11,965.34 $\pm 170.22$	10,726.97$\pm 493.52$	10,980.42$\pm 290.10$
layer = 2	Utility	$\mathbf{8537.65} \pm \mathbf{25.79}$	$8233.59 \pm 41.36$	$8461.69\pm 60.14$	$7681.31\pm 30.87$	$7890.54\pm 116.82$
layer = 4	Utility	16,360.63$\pm \mathbf{78.11}$	16,169.48$\pm 105.67$	16,217.54 $\pm 19.98$	14,363.06$\pm 14.72$	14,926.88$\pm 251.36$
step = 50	Utility	$\mathbf{7613.84} \pm \mathbf{25.55}$	$7293.24\pm 32.55$	$7296.38\pm 20.91$	$6755.67\pm 52.91$	$6984.32\pm 103.55$
step = 200	Utility	24,339.01$\pm \mathbf{69.32}$	24,101.43$\pm 73.49$	24,260.51$\pm 70.25$	20,847.72$\pm 84.13$	22,183.70$\pm 411.42$
task $\lambda =1$	Utility	$\mathbf{6354.41} \pm \mathbf{16.67}$	$6049.25\pm 43.57$	$5985.33\pm 29.46$	$5666.33\pm 15.80$	$5841.26\pm 81.34$
task $\lambda =3.5$	Utility	19,826.57$\pm \mathbf{25.89}$	19,286.34$\pm 135.23$	19,474.48$\pm 41.06$	17,493.73$\pm 68.05$	18,120.66$\pm 326.28$
worker = 6	Utility	12,737.38$\pm \mathbf{24.76}$	12,504.35$\pm 91.72$	12,321.40 $\pm 68.41$	11,412.98$\pm 28.43$	11,742.30$\pm 180.66$
worker = 10	Utility	13,139.97$\pm \mathbf{16.88}$	12,954.70$\pm 45.95$	12,970.57$\pm 38.78$	11,707.93$\pm 46.42$	12,064.51$\pm 196.44$

Note: Bold values indicate the best performance for each listed setting.

summarises the final evaluation under layer, episode length, task load, and worker-count perturbations. CReSCENT is the best method on utility in every listed setting. On the standard setting, its utility is 12,334.89, while EMU, ETA-PSI, MIMEx, and the external-only baseline obtain 12,003.16, 11,965.34, 10,726.97, and 10,980.42. The external-only comparison shows that the improvement is not merely the result of adding any bonus. It comes from combining structural representation, milestone-weighted pseudo counts, and layer-aware routing of the intrinsic signal. Under a deeper network with four layers, the utility margin remains visible, which supports the claim that structural representation and credit allocation are useful when the cross-layer path to a milestone grows longer.

Table 5. Statistical validation for default-setting utility.

Comparison	Mean Utility Gain	95% Confidence Interval	Wilcoxon p-Value
CReSCENT vs. EMU	331.73	[205.6, 457.9]	0.004
CReSCENT vs. ETA-PSI	369.55	[216.3, 522.8]	0.003
CReSCENT vs. MIMEx	1607.92	[1268.4, 1947.5]	<0.001
CReSCENT vs. external-only	1354.47	[1003.2, 1705.7]	<0.001

reports the statistical validation on the default setting. The confidence interval is computed for the paired utility difference between CReSCENT and each baseline across ten independent seeds. The p-value is obtained from a two-sided paired Wilcoxon signed-rank test. CReSCENT is significantly better than all four baselines at the 0.05 level. The smallest margin is against EMU, yet the confidence interval remains positive, which supports the conclusion that the improvement is not caused by a single favourable run.

The layer perturbation tests whether the method survives a longer chain of credit. A deeper network increases the delay between early dispatching and milestone reward. CReSCENT keeps a stable advantage because the macro feature vector explicitly records progress and bottleneck position, while the contrastive loss keeps structurally close states adjacent in the latent space. The episode-length perturbation tests whether the method handles short and long credit horizons. The short horizon requires quick exploration; the long horizon requires persistence. Pseudo-count decay supports both, because new regions receive strong early bonuses and repeated regions gradually stop receiving large bonuses.

Task-load perturbation changes the number of possible structural paths. When $\lambda =3.5$, the number of queue configurations rises and naive novelty estimates become noisy. CReSCENT remains strongest because its novelty is computed after structural encoding. Worker-count perturbation changes action-space size and collaboration complexity. The method remains stable because layer credit allocation scales the intrinsic signal according to contribution rather than assigning it uniformly.

5.4. Ablation

Figure 4 reports the ablation curves. Removing contrastive learning weakens the structural alignment of latent states. Removing online clustering eliminates the pseudo-count mechanism and makes the intrinsic signal less adaptive. Removing credit allocation broadcasts novelty uniformly across layers and increases interference. The full model shows faster startup and higher final utility, which confirms that the four modules form a closed loop rather than four independent add-ons.

5.5. Parameter Sensitivity and Representation Analysis

Figure 5 varies the intrinsic coefficient $\eta$. A small value makes the method close to the external-reward baseline and delays milestone discovery. A large value over-emphasises exploration and may slow final exploitation. The middle range used as default provides the best balance between structural coverage and external objective alignment.

Table 6. Sensitivity of the online clustering threshold $\rho$.

$\rho$	Number of Active Clusters	Utility	Milestone Hit Rate	Interpretation
0.20	94	12,110 ± 96	71.8%	Over-fragmented structural stages
0.30	71	12,240 ± 88	74.9%	Stable but still fine-grained
0.40	58	12,335 ± 80	78.6%	Default balance
0.50	43	12,302 ± 84	77.4%	Slightly merged bottleneck states
0.60	31	12,195 ± 91	74.2%	Under-separated milestone regions

Note: Bold values indicate the best performance in this sensitivity study.

reports the sensitivity of the clustering threshold $\rho$. The threshold is applied to the unit-normalised latent vectors after contrastive encoding. Very small values fragment one structural stage into many clusters and make the pseudo-count reset too frequently. Very large values merge bottleneck states with different milestone proximity and reduce the usefulness of novelty estimation. The default value $\rho =0.40$ gives the best utility and the highest milestone hit rate in this sweep.

Table 7. Sensitivity of cross-layer credit-allocation coefficients.

Coefficient Setting $({\mathit{a}}_{\mathit{m}},{\mathit{a}}_{\mathit{b}},{\mathit{a}}_{\mathit{u}})$	Utility	Milestone Hit Rate	Observation
(0.50, 0.30, 0.20)	12,334.89 ± 79.84	78.6%	Default milestone-first balance
(0.40, 0.40, 0.20)	12,291.36 ± 86.17	77.5%	More bottleneck oriented
(0.60, 0.25, 0.15)	12,318.04 ± 82.75	78.1%	Stronger milestone emphasis
(0.33, 0.33, 0.34)	12,192.77 ± 104.63	75.9%	Uniform credit is less selective
(0.45, 0.20, 0.35)	12,220.45 ± 97.31	76.8%	Over-rewards utilisation

Note: Bold values indicate the best performance in this sensitivity study.

reports the sensitivity of the credit-allocation weights. The method remains stable when the three weights are perturbed around the default setting. The milestone-heavy setting is slightly best, which is consistent with the delayed-reward objective, while the uniform setting is weaker because it does not distinguish milestone progress from generic utilisation.

Figure 6 visualises latent trajectories. The revised figure caption states the method in each panel, the axis meaning, and the colour semantics. The colour encodes the normalised within-episode step index, and larger highlighted points indicate states near milestone activation. Without intrinsic learning, the state samples remain concentrated near a limited region and show weak structural expansion. Pure intrinsic learning spreads widely but is less aligned with milestone value. CReSCENT produces a trajectory that covers diverse regions while retaining a coherent progression, which is the desired behaviour under delayed reward. We also repeated the latent-space inspection with UMAP and PaCMAP.

Table 8. Projection diagnostics for latent trajectories. Higher values indicate better milestone-stage organisation.

Projection	Neighbour Trustworthiness	Milestone-Neighbour Fraction	Stage Monotonicity
t-SNE	0.91	0.78	0.74
UMAP	0.89	0.76	0.72
PaCMAP	0.90	0.77	0.73

shows the same qualitative conclusion across all three projection tools, so the interpretation is not tied only to t-SNE.

5.6. Discussion

CReSCENT is most useful when reward is delayed because of structural completion rather than because of random observation sparsity. In such settings, the question is not only whether a state is new. The question is whether the state represents a new form of cross-layer progress. This is why the macro feature map and contrastive encoder matter. They prevent the novelty module from rewarding superficial state differences and push it toward milestone-relevant distinctions.

The method also clarifies the role of intrinsic reward in industrial scheduling. Intrinsic reward should not be a permanent second objective. It should be a temporary scaffold that increases the probability of seeing useful external events. The pseudo-count form supports this role by decaying within revisited clusters. Once the policy repeatedly reaches a structural region, the novelty signal falls, and the external milestone reward becomes the dominant training signal.

The main limitation is that the macro feature map uses domain knowledge. This is appropriate for the studied industrial-network setting, where the process structure is known, but a fully general system would need automatic discovery of bottleneck and milestone descriptors. A second limitation is that online clustering introduces hyperparameters such as the radius and the cluster budget. The sensitivity analysis indicates stable behaviour in the tested range, but adaptive radius selection would reduce deployment effort.

The feature construction strategy is expected to transfer when a new industrial network can expose the same family of aggregate descriptors, such as queue pressure, resource utilisation, buffer blockage, and milestone proximity. It is less direct when the network contains non-serial routing, re-entrant processing, or flexible product recipes. In those settings, the macro map should be extended from a fixed feature vector to a graph encoder that consumes the current process graph and produces layer or node embeddings. The present results therefore support the value of structural exploration, while the exact feature engineering should be adapted to the process topology.

The simulator used in the experiments is part of the project environment, so the full code repository is not included in the submission package. To improve reproducibility, the revision gives the transition graph, task-entry rule, processing and transfer rule, milestone trigger, reward formula, utility definition, random-seed protocol, and baseline access conditions. These details are sufficient to reconstruct a simplified benchmark with the same delayed-reward mechanism. The data and implementation files can be provided by the corresponding author upon reasonable request, subject to project-sharing constraints.

6. Conclusions

We formulated delayed-reward multi-layer industrial scheduling as TD-MIC-MDP and showed that external reward alone can fail to distinguish promising trajectory prefixes before milestone activation. CReSCENT addresses this problem by learning structural representations, estimating milestone-weighted novelty through online pseudo counts, and allocating the resulting intrinsic signal to layers according to milestone contribution. The revised experiments show consistent utility gains under changes in layer count, episode length, task load, and worker count, and the added external-only baseline verifies that the gain is not merely caused by adding generic exploration pressure. Statistical tests, threshold sensitivity, credit-weight sensitivity, and projection diagnostics further support the robustness of the conclusion. Future work should automate macro feature discovery and combine structural novelty with uncertainty-aware value estimation.