Synergistic Evolutionary Optimization with Reinforcement Learning for Multi-Objective Energy-Efficient Hybrid Flow Shop Scheduling

Abstract

The Energy-Efficient Hybrid Flow Shop Scheduling Problem poses a significant multi-objective optimization challenge, necessitating the simultaneous minimization of conflicting objectives: Total Tardiness, Total Energy Cost, and Carbon Trading Cost. The Non-dominated Sorting Genetic Algorithm II (NSGA-II) is a classic algorithm in the field of multi-objective optimization. However, this algorithm frequently lacks the adaptive capability required to navigate high-dimensional solution spaces, often trapping the search in local optima, particularly when constrained by practical energy states of heterogeneous machines. To address these complexities, this study proposes a hybrid algorithm, named QGN, integrating Q-learning, the Grey Wolf Optimizer (GWO), and the NSGA-II. Specifically, QGN algorithm integrates NSGA-II for robust diversity maintenance with GWO for high-precision intensification. Unlike static hybrid methods, QGN employs a Q-learning agent as an adaptive controller to dynamically balance global exploration and local refinement, providing a theoretically grounded response to the rugged search landscape created by machine heterogeneity. Comprehensive experimental validation across diverse production scenarios confirms that QGN significantly outperforms baselines, including NSGA-II, Jaya, and Multi-Objective Evolutionary Algorithm based on Decomposition (MOEA/D), as well as the state-of-the-art Q-learning- and GVNS-driven NSGA-II (QVNS) algorithm, in terms of both convergence and diversity. The results indicate that the proposed algorithm yields superior solution dominance, generates a substantially larger set of non-dominated solutions, and maintains a more uniform distribution along the Pareto front.

1. Introduction

The rise of Industry 4.0 has fundamentally reshaped the manufacturing industry. Modern production systems now balance traditional efficiency targets with urgent sustainability goals [1]. This shift is driven by global pressure to mitigate climate change and strictly control industrial carbon emissions [2]. Consequently, manufacturing enterprises are moving away from purely time-based scheduling toward Energy-Efficient Scheduling (EES) [3]. This approach treats energy consumption not merely as a fixed cost but as a variable that can be optimized through intelligent decision-making [4].

In discrete manufacturing industries such as electronics and semiconductors, the Hybrid Flow Shop (HFS) [5] remains a dominant production layout. HFS environments are characterized by multiple processing stages where at least one stage contains parallel machines [6]. Scheduling in this environment involves complex sub-problems: sequencing jobs across stages and assigning specific tasks to parallel machines. This complexity makes the HFS scheduling problem (HFSP) [7] a distinct NP-hard combinatorial challenge where finding an optimal solution requires exponential time as problem size increases.

Traditional research on HFSP largely focuses on minimizing the makespan [8]. However, as manufacturing paradigms shift towards sustainability, the pursuit of Green Manufacturing requires a multi-objective perspective [9]. The Energy-Efficient Hybrid Flow Shop Scheduling Problem (EEHFSP) [10] seeks to optimize conflicting objectives simultaneously: minimizing total tardiness (TT) to ensure service levels, while reducing Total Energy Cost (TEC) and Carbon Trading Cost (CTC) to meet environmental regulations. This simultaneous optimization creates a complex and high-dimensional solution space where improving one objective often degrades others, making the identification of an optimal trade-off exceptionally difficult. Although converting these criteria into a single economic indicator via convolution is a common simplification, it obscures intrinsic operational conflicts and restricts the exploration of high-dimensional trade-offs.

Beyond these objective conflicts, a critical gap in the existing EEHFSP literature is the oversimplification of machine characteristics. Many studies assume parallel machines at the same stage are identical in speed and power [11]. In realistic industrial environments, factory floors often contain unrelated parallel machines with heterogeneous performance profiles due to differing models, ages, or wear states [12]. Furthermore, machines consume energy differently across various states, such as processing, idle, setup, and shutdown [13]. Ignoring this machine heterogeneity leads to theoretical models that fail to reflect actual shop floor energy dynamics, resulting in schedules that are unimplementable in practice [14]. Incorporating these realistic constraints significantly increases the problem’s computational complexity, rendering simple heuristic adjustments insufficient.

Solving high-dimensional, multi-objective EEHFSP instances poses significant methodological difficulties. Exact methods like Mixed-Integer Linear Programming are computationally prohibitive for large-scale industrial applications [15]. Consequently, metaheuristics such as the Genetic Algorithm (GA) and Particle Swarm Optimization (PSO) have become the standard solvers [16]. The NSGA-II is particularly favored for its effectiveness in multi-objective optimization [17]. However, relying on a single static search logic often proves insufficient when addressing specific, highly constrained multi-objective problems. Traditional metaheuristics often suffer from premature convergence and lack the adaptability required to navigate the rugged landscapes of complex energy scheduling problems [18]. Specifically, NSGA-II relies on fixed genetic operators that cannot adapt to the changing landscape of the optimization process, often causing the population to stagnate in local optima when facing the heterogeneous energy constraints identified above.

To mitigate the aforementioned shortcomings, the integration of Machine Learning (ML) into metaheuristics has emerged as a powerful research direction [19]. Specifically, Reinforcement Learning (RL) [20] offers a robust mechanism for online algorithm control. By modeling the optimizer as an agent that learns from its search history, RL can dynamically adjust search strategies, such as Adaptive Operator Selection (AOS) [21], to suit the current optimization state. This allows the algorithm to autonomously balance global exploration and local exploitation, significantly enhancing search efficiency without relying on static, manually tuned parameters [22]. Recent surveys indicate that hybridizing RL with evolutionary computation (ERL) [23] is a promising pathway for solving complex sequential decision-making and optimization tasks.

To surmount the limitations of conventional metaheuristics in handling the multi-objective EEHFSP, this study presents algorithmic framework named QGN. Unlike existing hybrid approaches that rely on static parameter settings or fixed operator probabilities, which often fail to adapt to the rugged search landscapes of heterogeneous machine environments, our approach introduces a dynamic control mechanism. While NSGA-II serves as a robust engine for maintaining a well-distributed Pareto front, strictly relying on its stochastic genetic variation limits the algorithm’s ability to locate high-precision optima in complex local regions. To address this specific deficiency where traditional methods suffer from premature convergence, the proposed framework embeds the GWO as a “point-wise” intensifier. This mechanism leverages the leadership of elite wolves to guide the population toward higher-quality solution regions, effectively complementing the global diversity maintained by NSGA-II with the local precision of swarm intelligence. Furthermore, by integrating a Q-learning agent, the QGN framework resolves the rigidity of manual parameter tuning found in previous works, enabling autonomous adaptation to the optimization state. The main contributions of this paper are as follows:

• This framework is constructed by embedding the exploration mechanisms of GWO into the NSGA-II structure. This parallel dual-path mechanism effectively mitigates the trade-off between global search and local exploitation, preventing the premature convergence often observed in conventional methods.
• Within this architecture, an intelligent decision module governed by Q-learning is developed to dynamically regulate genetic operators and search parameters. By perceiving real-time population states, this mechanism overcomes the rigidity of fixed evolutionary strategies through data-driven decision-making.
• Finally, the efficacy of the proposed approach is substantiated through extensive comparative experiments and ablation studies. Results confirm its superior convergence and diversity in simultaneously optimizing total tardiness, energy consumption, and carbon trading costs against standard baselines.

The article is organized to provide a logical progression from theoretical foundations to empirical analysis. Section 2 synthesizes related work to identify research gaps, while Section 3 outlines preliminary algorithmic concepts. Section 4 mathematically formulates the EEHFSP, detailing objective functions and constraints. The architecture of the QGN algorithm is elaborated in Section 5. Subsequently, Section 6 presents the experimental design and analyzes the comparative results. The study concludes in Section 7 with a summary of the work and future outlook.

2. Related Work

The EEHFSP involves complex trade-offs between time-based efficiency and environmental sustainability. Recent methodologies have shifted from static mathematical programming to adaptive, learning-driven optimization frameworks capable of handling multi-objective conflicts in dynamic manufacturing environments.

2.1. Optimization Objectives in EEHFSP

Modern scheduling models must simultaneously address production efficiency and energy consumption constraints, such as time-of-use electricity tariffs and carbon emissions. Zhang et al. [24] proposed a decomposition-based evolutionary algorithm for hybrid flow shops, demonstrating that breaking down complex energy objectives into sub-problems effectively balances makespan and total energy cost. While decomposition effectively handles static trade-offs, dynamic production environments introduce further complexities. Wang et al. [25] argued that static scheduling fails to capture real-time energy fluctuations in low-carbon environments, necessitating game-theoretic approaches to handle multi-time period constraints. Catanzaro et al. [26] provided a comprehensive survey on job scheduling under Time-of-Use energy tariffs, establishing that dynamic electricity pricing is a fundamental objective in modern sustainable manufacturing.

To meet stringent environmental regulations, optimization objectives have expanded beyond simple energy costs. Wu et al. [27] introduced specific energy-saving measures (ESM) into flexible job shop scheduling, validating that explicit energy objectives reduce consumption without severely degrading production time. To explicitly address machine energy states, Lee et al. [28] investigated idle time control and capacity adjustments under dynamic electricity pricing, laying the theoretical groundwork for the shutdown/standby mechanisms used in contemporary energy-saving strategies. Furthermore, Jiang et al. [29] highlighted that carbon emission constraints must be treated as distinct optimization objectives rather than secondary constraints to ensure compliance with industrial standards. Moreover, the scope of energy consumption in modern factories extends beyond processing machines to include auxiliary logistics. In distributed manufacturing contexts, Liu et al. [30] integrated crane transportation constraints into green scheduling, establishing that auxiliary resources significantly impact the total energy footprint. Ghorbanzadeh et al. [31] expanded the scope of energy-aware flow shop scheduling by incorporating renewable energy constraints and setup times, highlighting the necessity of multi-dimensional environmental objectives.

Despite these advancements, the existing literature rarely addresses the simultaneous optimization of Total Tardiness, Total Energy Cost, and Carbon Trading Cost within a unified framework. Most current models oversimplify the manufacturing environment by assuming identical parallel machines, thereby failing to capture the economic pressure of carbon quotas in realistic heterogeneous machine environments. This study bridges this gap by formulating a comprehensive multi-objective model that explicitly accounts for machine heterogeneity and complex carbon trading constraints.

2.2. Heuristic and Evolutionary Algorithms

Metaheuristics such as NSGA-II and GWO have become standard solvers for scheduling problems due to their effectiveness. NSGA-II is favored for its multi-objective capabilities, while Swarm Intelligence algorithms like GWO offer strong local exploitation. For example, Lu et al. [32] proposed a multi-objective cellular grey wolf optimizer for the hybrid flow shop scheduling problem, demonstrating the potential of GWO in handling complex manufacturing constraints. While metaheuristics are the standard for NP-hard scheduling problems, conventional algorithms often struggle with high-dimensional solution spaces and premature convergence. However, relying solely on standard metaheuristics often proves insufficient for large-scale instances. Shao et al. [33] utilized a multi-objective evolutionary algorithm with multiple neighborhood searches, yet noted that fixed operator selection often leads to stagnation in local optima. Similarly, Zuo et al. [34] applied a multi-population artificial bee colony algorithm to improve diversity, but found parameter adaptability to be a bottleneck when scaling to larger problem instances. Penn et al. [35] systematically analyzed the computational complexity of scheduling under TOU tariffs, demonstrating the indispensable role of advanced heuristic approaches over exact methods when dealing with highly volatile energy cost objectives.

Swarm intelligence algorithms, such as the GWO, offer strong local exploitation but lack inherent mechanisms for global exploration in discrete spaces. Tao et al. [36] employed a discrete imperialist competitive algorithm for resource-constrained flow shops, finding that static assimilation strategies limit the discovery of novel solutions. Recent efforts focus on algorithm-specific enhancements. For instance, Li et al. [37] developed an improved Jaya algorithm that incorporates transportation and setup times, effectively reducing makespan in flexible environments. However, Gu et al. [38] observed that even improved cuckoo search algorithms require hybrid mechanisms to balance the exploration-exploitation trade-off effectively in multi-objective energy-efficient scenarios. Recently, Li et al. [39] designed an Improved NSGA-II (INSGA-II) combined with the IDNEH heuristic to address the energy-efficient distributed blocking flow shop scheduling problem (EEDBFSP), aiming to minimize makespan and total energy consumption under blocking constraints.

Although specific enhancements have improved solution quality, traditional metaheuristics still predominantly rely on fixed operators and static assimilation strategies. They lack the online adaptability required to navigate the rugged landscapes of the multi-objective EEHFSP, often leading to premature convergence. Unlike these static approaches, the proposed QGN algorithm introduces a dynamic dual-path mechanism that leverages swarm intelligence for intensification while retaining genetic diversity, ensuring a more robust balance between exploration and exploitation.

2.3. Hybrid Evolutionary Reinforcement Learning

RL addresses the rigidity of metaheuristics by enabling agents to learn optimal policies through interaction with the production environment. Ni et al. [40] applied Proximal Policy Optimization (PPO) to large-scale hybrid flow shops, demonstrating that learning-based methods significantly outperform rule-based heuristics in handling high-dimensional state spaces. Gil et al. [41] further utilized Deep Reinforcement Learning (DRL) for material scheduling, verifying that agents can autonomously adapt to dynamic constraints without prior domain knowledge.

The adaptability of RL allows for robust decision-making under uncertainty. Liu et al. [42] combined agent-based simulation with RL to manage multi-skilled worker constraints, showing superior robustness against fatigue factors compared to traditional solvers. Zhang et al. [43] extended this by applying meta-reinforcement learning to address learning and forgetting effects, proving that adaptive mechanisms effectively mitigate processing time uncertainty. Additionally, Zhao et al. [44] introduced a heterogeneous graph neural network combined with DRL, capturing topological dependencies between machines and jobs better than standard vector-based state representations. Lin et al. [45] specifically addressed reentrant hybrid flow shops using DRL, confirming that learning agents can optimize stocker placement and scheduling simultaneously.

RL agents can learn to adjust search strategies dynamically based on population feedback. For instance, Peize Li et al. [46] developed a method combining Q-learning with a GVNS-driven NSGA-II, where RL is used to guide the local search process for energy-efficient scheduling. Internationally, Momenikorbekandi et al. [47] successfully applied advanced Q-learning models to dynamically optimize complex job shop schedules, demonstrating its efficacy in managing production variations. Integrating Evolutionary Algorithms (EAs) with RL, termed ERL, combines global search capabilities with rapid decision-making. Cai et al. [48] integrated Q-learning with a shuffled frog-leaping algorithm, where the RL agent dynamically selects search strategies based on population feedback, significantly improving convergence. Similarly, Wang et al. [49] employed a cooperative memetic algorithm with a learning-based agent, demonstrating enhanced performance in distributed scheduling tasks by allowing agents to share knowledge.

This hybridization extends to complex multi-objective problems. Jia et al. [50] proposed a Q-learning-driven multi-population memetic algorithm for distributed assembly flow shops, utilizing RL to adjust preventive maintenance schedules dynamically. Xi and Lei [51] combined Q-learning with teaching-learning-based optimization, effectively handling fuzzy processing times in two-stage hybrid flow shops. In terms of operator selection, Li et al. [52] demonstrated that double Deep Q-Networks (DQN) can co-evolve with heuristic operators to solve distributed heterogeneous scheduling, validating the synergy between learning and evolution. Similarly, Zhang et al. [53] proposed a co-evolutionary algorithm driven by a Double Deep Q-Network (DDQN) for the dynamic energy-efficient hybrid flow shop scheduling problem with transportation constraints, utilizing the agent to autonomously select optimal local search operators. Cui et al. [54] confirmed that RL-guided genetic algorithms outperform static counterparts in energy-aware production by learning optimal parameter configurations online. Furthermore, Deng et al. [55] utilized RL to estimate distribution algorithms for fuzzy distributed scheduling, balancing on-time delivery with energy costs. In a similar vein, Zhou et al. [56] developed a multi-objective reinforcement learning-based memetic algorithm for dynamic hybrid flow shop scheduling with energy storage systems, where Q-learning is employed to autonomously select the most suitable local search operators based on the evolutionary state. Wang et al. [57] explored independent double DQN for multi-agent scheduling, providing a framework for decentralized decision-making in batch processing. Finally, Qin et al. [58] utilized Q-learning to control the iterated greedy algorithm in blocking flow shops, while Zhao et al. [59] applied cooperative water wave optimization with RL, both supporting the conclusion that dynamic control of algorithmic components yields superior solution quality in energy-efficient scheduling.

While existing hybrid approaches have successfully utilized Q-learning for operator selection, they predominantly rely on a single search logic, effectively only tuning the parameters of one base algorithm. They often lack a synergistic framework that actively combines distinct search paradigms. The QGN algorithm proposed in this study addresses this limitation by implementing a dual-path mechanism. By utilizing the Q-learning agent to dynamically arbitrate between the vector-based exploration of GWO and the genetic evolution of NSGA-II, our approach achieves a theoretically superior equilibrium between diversity and convergence compared to single-logic hybrids.

3. Preliminary

3.1. NSGA-II

The NSGA-II proposed by Deb et al. [17] is a classic evolutionary algorithm for multi-objective optimization, widely applied in engineering optimization and scheduling. Its core principle simulates biological evolution, utilizing non-dominated sorting and crowding distance calculation to screen a uniformly distributed, well-converged Pareto optimal solution set from the population, thereby balancing multi-objective conflicts and approximating the true Pareto frontier.

NSGA-II stratifies the population via non-dominated sorting: an individual “dominates” another if it performs no worse across all objectives and outperforms in at least one. The process involves counting $n_p$ (number of individuals dominating the current individual) and $S_p$ (set of individuals dominated by the current individual) for each individual, classifying those with $n_p=0$ into the first non-dominated layer ($F_1$), and repeating the process for remaining individuals after removing each layer. This method prioritizes superior multi-objective trade-off performance, with an optimized time complexity of $O\left(M{N}^2\right)$ (where M is the number of objectives and N is the population size).

To maintain population diversity, NSGA-II introduces the crowding distance metric, which quantifies the sparsity of individuals within their frontier layer. For each non-dominated layer, boundary individuals (with maximum or minimum objective values) are assigned infinite crowding distance to ensure retention; intermediate individuals’ crowding distance is calculated as:

$$d_p=d_p+{\displaystyle \frac{f_j(p+1)-f_j(p-1)}{f_j^{\max }-f_j^{\min }}},$$

(1)

where $f_j$ denotes the j-th objective function, $f_j(p+1)$ and $f_j(p-1)$ are the j-th objective values of adjacent individuals in the sorted list, and $f_j^{\max }/f_j^{\min }$ are the maximum and minimum values of the j-th objective in the current layer. A larger crowding distance indicates a sparser surrounding solution space, enhancing the diversity of the Pareto frontier.

NSGA-II’s selection strategy combines elitism preservation and tournament selection. The elitism preservation mechanism merges the parent population ($P_t$) and offspring population ($Q_t$) into a mixed population $R_t$ (size $2N$), performs non-dominated sorting, sequentially selects individuals from higher layers, and retains those with higher crowding distances if a layer exceeds the target population size (N), thus preserving high-quality solutions. Tournament selection randomly selects individuals, comparing their dominance relationships and crowding distances to select superior parents, guiding the population’s evolutionary direction.

3.2. GWO

The GWO proposed by Mirjalili et al. [60] is a swarm intelligence optimization algorithm widely used in function optimization and engineering design, characterized by a simple structure and strong exploration capability. Its core concept simulates the social hierarchy and predation mechanism of grey wolf packs, realizing global optimal solution search through group collaboration.

GWO adopts a strict social hierarchy of grey wolves: $\alpha$-wolves represent the current optimal solution and make hunting decisions; $\beta$-wolves (suboptimal solutions) assist $\alpha$-wolves in strategy adjustment; $\delta$-wolves execute specific tasks following upper-level wolves; and $\omega$-wolves follow the guidance of $\alpha$, $\beta$, and $\delta$-wolves to maintain population coordination. This hierarchy guides the population toward high-quality solution regions.

A key parameter of GWO is the convergence factor a, which decreases linearly from 2 to 0 with the number of iterations. A larger a (close to 2) in the early iteration stage enhances global exploration, while a smaller a (close to 0) in the later stage strengthens local exploitation, effectively balancing exploration and exploitation capabilities.

The position update of GWO relies on two core formulas: the encircling prey formula and the weighted average of guiding wolves’ positions. The encircling prey is calculated as:

$$D=|C·X_p\left(t\right)-X\left(t\right)|,$$

(2)

$$X(t+1)=X_p\left(t\right)-A·D,$$

(3)

where $X\left(t\right)$ is the current individual position, $X_p\left(t\right)$ is the position of $\alpha$, $\beta$, or $\delta$-wolf, $A=2a·r_1-a$ and $C=2·r_2$ ($r_1,r_2\in [0,1]$ are random vectors). The absolute value of A regulates exploration intensity ($\left|A\right|>1$ for global exploration, $\left|A\right|<1$ for local exploitation), and C introduces random disturbances to avoid local optima. Based on the positions of $\alpha$, $\beta$, and $\delta$-wolves, the new individual position is generated by weighted average:

$$X(t+1)={\displaystyle \frac{X_1(t+1)+X_2(t+1)+X_3(t+1)}{3}},$$

(4)

where $X_1(t+1)$, $X_2(t+1)$, and $X_3(t+1)$ are positions guided by $\alpha$, $\beta$, and $\delta$-wolves, respectively, ensuring inheritance of high-quality genes and expansion of exploration scope.

3.3. Q-Learning

Q-learning, proposed by Watkins and Dayan [61], is a model-free RL algorithm that does not require prior knowledge of the environmental model, widely applied in dynamic scheduling, path planning, and intelligent control. Its core idea is to enable an agent to learn the state-action value function (Q-function) through continuous interaction with the environment, thereby optimizing decision-making strategies.

In the Q-learning framework, the environment’s current condition is defined as a “state” (s), and the agent’s executable operations are “actions” (a). The Q-function $Q(s,a)$ quantifies the expected long-term reward of performing action a in state s, with a higher value indicating a more optimal action. The core of Q-learning lies in the Q-value update mechanism, which follows the temporal difference (TD) learning rule:

$$Q(s,a)\leftarrow Q(s,a)+\alpha \left[r+\gamma \underset{a^{\prime }}{\max }Q(s^{\prime },a^{\prime })-Q(s,a)\right],$$

(5)

where $\alpha$ (learning rate, typically 0.1–0.5) controls the update amplitude to avoid noise-induced oscillations, $\gamma$ (discount factor, typically 0.8–0.95) weights the importance of future rewards, r is the immediate reward feedback from the environment, and ${\max }_{a^{\prime }}Q(s^{\prime },a^{\prime })$ denotes the maximum expected reward of the optimal action in the new state $s^{\prime }$.

To balance exploration and exploitation, Q-learning adopts the $\epsilon$-greedy strategy. The agent selects the action with the highest current Q-value (exploitation) with probability $1-\epsilon$ ($\epsilon \in [0,1]$) and randomly selects an action (exploration) with probability $\epsilon$, avoiding premature strategy solidification and local optima.

The integration of these three methodologies is theoretically justified by the complementary nature of their search behaviors. NSGA-II provides the global exploration framework necessary for the combinatorial part of EEHFSP, while GWO provides a more concentrated search effort essential for optimizing conflicting energy-efficiency objectives. By embedding these into a reinforcement learning structure, the Q-learning agent overcomes the rigidity of static algorithm configurations. It autonomously selects the most effective operator based on the current population status, thereby actively adapting to the specific optimization landscape of the EEHFSP.

4. Problem Formulation

The production system of the HFSP comprises s processing stages ($k=\{1,2,\dots ,s\}$), where each stage is equipped with m identical parallel machines. The set of machines in stage k is denoted as $M_k=\{1,2,\dots ,m\}$, and the i-th machine in stage k is referred to as $m_{ik}$. The set of jobs to be scheduled is $J=\{1,2,\dots ,n\}$; each job j must pass through all stages sequentially following a fixed process route, and the operation $o_{jk}$ (i.e., the operation of job j in stage k) can be assigned to any machine in that stage for processing.

The benchmark processing time of operation $o_{jk}$ is $p_{jk}$; if assigned to machine i in stage k, the actual processing time is adjusted based on the machine’s operating parameters. Each job j has a specified due date $d_j$; if the total completion time $C_{js}$ of job j (i.e., completion time at the final stage) exceeds $d_j$, a tardiness time $T_j=\max (0,C_{js}-d_j)$ is incurred.

The core goal of scheduling is to optimize three objectives simultaneously: first, minimize the TT to ensure orders are delivered on time; second, reduce the TEC by leveraging TOU electricity pricing and machine shutdown strategies to cut down energy expenses; third, lower the CTC by controlling carbon emissions through carbon allowances and carbon prices. Meanwhile, the scheduling decision must comply with constraints such as fixed job process routes and machine resource allocation, ultimately generating a Pareto optimal scheduling scheme.

To align with real-world manufacturing scenarios and simplify model complexity, several assumptions are established. For production processes, job processing cannot be interrupted, and each machine can only handle one job at a time. Jobs follow a fixed process route without skipping or repeating any stages, and all jobs become available for scheduling at time 0 with no release delays.

In terms of machine states, machines operate in four distinct modes—processing, standby, restart, and shutdown—each with unique energy consumption characteristics. The processing state has the highest energy consumption, calculated as ${\lambda }_{ik}{p}_{ek}$, where ${\lambda }_{ik}$ is the speed-power conversion coefficient and $p_{ek}$ is the benchmark processing power of stage k. The standby state consumes less power, quantified as $s_{eik}$, while the restart state—which only occurs when a machine restarts after being shut down—uses power equal to $r_{eik}$ and lasts for a fixed duration $t_{reset}$. The shutdown state has zero energy consumption and is activated only when the machine’s idle time exceeds the “break-even time” $T_{Bik}$—a critical threshold where the energy consumed during restart equals that of keeping the machine on standby.

For time and energy-related factors, electricity pricing follows a TOU mechanism, with $f\left(t\right)$ representing the electricity price at time t and significant differences between peak, flat, and off-peak periods. Carbon emissions originate solely from machine energy consumption, with a fixed carbon emission factor $\mu$. The carbon trading system adopts a “baseline quota system”: if emissions exceed the carbon allowance EA, costs are incurred at the carbon price $C_p$, while no revenue is generated for emissions below the allowance.

Regarding scheduling rules, job transportation time, machine failures, and maintenance time are not considered. No priority constraints are imposed on jobs, so the processing order of jobs on the same machine can be flexibly adjusted. Additionally, machine restart cannot be interrupted, and the machine must immediately enter either the processing or standby state once the restart process is complete.

The notation involved in the mathematical model are defined in Table 1.

Table 1. The notation.

Notation	Definition
J	Job set, $J=\{1,2,\dots ,n\}$
K	Processing stage set, $K=\{1,2,\dots ,s\}$
$M_k$	Machine set in the k-th processing stage, $M_k=\{1,2,\dots ,m\}$
T	Time period set, $T=\{1,2,\dots ,t_{\max }\}$ (where $t_{\max }$ is the maximum scheduling horizon)
$d_j$	Due date of job j (unit: hour)
$p_{jk}$	Benchmark processing time of operation $o_{jk}$ (job j in stage k, unit: hour)
$v_{ik}$	Processing speed of machine $m_{ik}$ (stage k, unit: unit/hour)
${\lambda }_{ik}$	Speed-power conversion coefficient corresponding to $v_{ik}$
$p_{ek}$	Baseline processing power of machine $m_{ik}$ (stage k, unit: kW)
$r_{eik}$	Restart power of machine $m_{ik}$ (stage k, unit: kW)
$s_{eik}$	Standby power of machine $m_{ik}$ (stage k, unit: kW)
$f\left(t\right)$	Electricity price during time period t (unit: CNY/MWh)
$t_{\mathrm{reset}}$	Duration of machine restart (constant, unit: hour)
B	A sufficiently large positive integer
$C_p$	Carbon price (unit: CNY/ton CO2)
$EA$	Carbon emission allowance (unit: ton CO2)
$\mu$	Carbon emission factor per unit electricity (constant, unit: ton CO2/MWh)
$x_{jktik}$	Binary variable: 1 if $o_{jk}$ is processed on $m_{ik}$ during time t; 0 otherwise
$y_{jktik}$	Binary variable: 1 if $o_{jk}$ starts processing on $m_{ik}$ during time t; 0 otherwise
$z_{j{i}^{\prime }k}$	Binary variable: 1 if job j is processed immediately after job $i^{\prime }$ on $m_{ik}$; 0 otherwise
$u_{ikt}$	Binary variable: 1 if $m_{ik}$ is in processing/standby state during time t; 0 otherwise
$w_{ikt}$	Binary variable: 1 if $m_{ik}$ is in restart/standby state during time t; 0 otherwise
$E_{ikt}$	Total energy consumption of $m_{ik}$ during time t (unit: MWh)
$S_{jk}$	Start time of job j in stage k (unit: hour)
$C_{jk}$	Completion time of job j in stage k (unit: hour)
$C_{js}$	Completion time of job j in the final stage (total completion time)
$T_j$	Tardiness time of job j, $T_j=\max (0,C_{js}-d_j)$

The mathematical model for the EEHFSP, encompassing the optimization objectives and the necessary operational constraints, is formulated as follows:

$$\min TT=\sum _{j=1}^n{T}_j=\sum _{j=1}^n\max (0,C_{js}-d_j),$$

(6)

$$\min TEC=\sum _{k=1}^s\sum _{i=1}^m\sum _{t=1}^{t_{\max }}{E}_{ikt}·f\left(t\right),$$

(7)

$$\min CTC=C_p·\max \left(0,\mu ·\sum _{k=1}^s\sum _{i=1}^m\sum _{t=1}^{t_{\max }}{E}_{ikt}-EA\right),$$

(8)

$$E_{ikt}=s_{eik}·u_{ikt}·\Delta t+r_{eik}·w_{ikt}·t_{\mathrm{reset}}+{\lambda }_{ik}{p}_{ek}·x_{jktik}·\Delta t,$$

(9)

$$\sum _{j=1}^n{x}_{jktik}\le 1,\forall i\in M_k,k\in K,t\in T,$$

(10)

$$\sum _{t=1}^{t_{\max }}{y}_{jktik}=1,\forall i\in M_k,j\in J,k\in K,$$

(11)

$$\sum _{j=1}^n\sum _{i^{\prime }=0,i^{\prime }\ne j}^n{z}_{j{i}^{\prime }k}=1,\forall i\in M_k,k\in K,$$

(12)

$$y_{jktik}\le u_{ikt},\forall i\in M_k,j\in J,k\in K,t\in T,$$

(13)

$$w_{ikt}\le u_{ikt},\forall i\in M_k,k\in K,t\in T,$$

(14)

$$v_{ik}={\displaystyle \frac{p_{jk}}{{\sum }_{t=1}^{t_{\max }}{x}_{jktik}·\Delta t}},\forall i\in M_k,j\in J,k\in K,$$

(15)

$$C_{jk}=S_{jk}+\sum _{t=1}^{t_{\max }}{x}_{jktik}·\Delta t,\forall i\in M_k,j\in J,k\in K,$$

(16)

$$S_{j(k+1)}\ge C_{jk},\forall j\in J,k=1,2,\dots ,s-1,$$

(17)

$$C_{i^{\prime }k}+B·(1-z_{j{i}^{\prime }k})\le S_{jk},\forall i\in M_k,j\ne i^{\prime }\in J\cup \left\{0\right\},k\in K,$$

(18)

$$C_j=C_{js},\forall j\in J,$$

(19)

$$T_j\ge C_{js}-d_j,\forall j\in J,$$

(20)

$$u_{ikt}=0\mathrm{if}\mathrm{idle}\mathrm{time}\mathrm{of}{m}_{ik}>T_{Bik},\forall i\in M_k,k\in K,t\in T,$$

(21)

$$x_{jktik},y_{jktik},z_{j{i}^{\prime }k},u_{ikt},w_{ikt}\in \{0,1\},\forall i\in M_k,j\ne i^{\prime }\in J\cup \left\{0\right\},k\in K,t\in T.$$

(22)

The mathematical formulation above details the optimization goals and physical constraints of the scheduling problem. Equations (6)–(8) define the three minimization objectives. Specifically, Equation (6) calculates the TT to ensure timely delivery. Equation (7) computes the TEC by integrating the power consumption across different machine states with the TOU electricity tariffs. Equation (8) determines the CTC under a baseline quota system, where costs are incurred only when the actual emissions exceed the pre-assigned allowance ($EA$). While tardiness and carbon emissions can theoretically be estimated economically and convoluted into a unified cost function, they are maintained as independent objectives in this model. A single scalarized objective relies heavily on predefined economic weights that are highly volatile in real-world markets. By evaluating them separately, the model preserves a multidimensional trade-off space, ensuring that potentially high-quality solutions with exceptional performance in one specific dimension are not prematurely discarded during the optimization process.

Regarding energy dynamics and machine states, Equation (9) aggregates the total energy consumption ($E_{ikt}$) derived from processing, standby, and restart activities. This is complemented by Equation (21), which enforces an energy-saving shutdown strategy triggered when a machine’s idle time surpasses the break-even threshold ($T_{Bik}$). The logical consistency between machine states, processing speeds, and job assignments is maintained by Equations (13)–(15).

The feasibility of the scheduling scheme is governed by the remaining constraints. Equations (10)–(12) and Equation (22) ensure resource validity, stipulating that each machine processes only one job at a time, every job follows a continuous route, and decision variables remain binary. Finally, temporal logic is strictly enforced through Equations (16)–(20). These constraints calculate cumulative completion times, guarantee that jobs proceed sequentially across stages without violation, and prevent overlapping operations on the same machine using a large positive integer B.

5. Methods

5.1. Framework of the QGN Algorithm

The QGN algorithm establishes a hierarchical closed-loop framework in Figure 1 that integrates reinforcement learning with dual optimization strategies to enhance search efficiency. Functioning as the high-level controller, a Q-learning agent regulates the optimization trajectory by analyzing real-time population states, thereby dynamically balancing exploration and exploitation intensities throughout the iterative process. Guided by historical value estimates from the Q-table, the system selectively activates distinct operational pathways. One branch executes crossover and mutation operators to achieve fine-grained local refinement, while the parallel branch utilizes GWO-based position updates to facilitate coarse-grained global search and mitigate premature convergence.

Figure 1. Framework diagram of QGN algorithm.

Following the generation of candidate solutions, the framework consolidates offspring from these divergent paths into a unified pool. A selection mechanism based on non-dominated sorting and crowding distance calculations is then employed to filter high-quality individuals for the subsequent generation. The optimization cycle concludes by quantifying performance improvements across objectives into reward signals. These signals are backpropagated to update the decision policy, enabling the agent to iteratively refine its operator selection strategy according to specific problem states.

5.2. NSGA-II Base Module

The NSGA-II base module serves as the core multi-objective optimization framework of the QGN algorithm. It undertakes the tasks of population sorting, screening, and elitism preservation, providing a high-quality initial population foundation for GWO-based exploration and Q-learning-driven decision-making. Its design focuses on three key steps, including non-dominated sorting, crowding distance calculation, and elite selection, to ensure both the convergence of multi-objective optimization and the diversity of the solution set.

5.2.1. Encoding and Decoding

This study employs a permutation-based encoding scheme to solve the hybrid flow shop scheduling problem. A single integer sequence $\pi$ containing all job indices represents the processing priority. This streamlined structure eliminates the need for complex machine allocation variables and effectively reduces the search space.

The decoding procedure transforms this job sequence into a concrete schedule using a list scheduling rule. In the first stage, the algorithm processes jobs strictly according to the initial order of sequence $\pi$. The scheduler iterates through all available machines in the current stage and selects the specific machine that allows the earliest completion time for the assigned job. For every subsequent processing stage, the algorithm first reorders the sequence $\pi$ in non-decreasing order based on the completion times from the previous stage. This dynamic sorting mechanism ensures that earlier arriving jobs receive priority for processing resources. Once the sequence is updated, the scheduler assigns each job to the machine that minimizes its completion time in the current stage. This greedy strategy applies iteratively across all stages until the entire schedule is generated. The specific process is shown in Algorithm 1.

Algorithm 1 Decoding Procedure

1: Input: Job permutation $\pi$   2: Output: A feasible schedule   3: for $j=1$ to n do   4:       Select the machine in $M_1$ that allows the earliest completion for job $\pi \left(j\right)$   5:       Update $C_{j,1}$   6: end for   7: for $k=2$ to s do   8:       Reorder $\pi$ by sorting jobs in non-decreasing order of $C_{j,k-1}$   9:       for $j=1$ to n do 10:             Select the machine in $M_k$ that allows the earliest completion for job $\pi \left(j\right)$ 11:             Update $C_{j,k}$ 12:       end for 13: end for

5.2.2. Initialization

The initialization process generates the starting population by combining heuristic guidance with stochastic diversity. The algorithm first employs an improved NEH heuristic to construct high-quality seed solutions tailored to the three optimization objectives. It calculates the total processing time for every job and sorts the entire job set in descending order to form a priority list. For each objective, the constructor selects the first two jobs from this list to generate an initial partial sequence that minimizes the specific target value. The algorithm then iterates through the remaining jobs in the sorted list one by one. In each step, the scheduler inserts the current job into the position within the existing sequence that yields the most favorable objective value. These three heuristic-driven solutions serve as elite seeds to guide the initial search. The algorithm subsequently fills the remainder of the population by generating random job permutations until the set cardinality reaches the specified limit. This hybrid strategy ensures the algorithm begins with strong candidates while maintaining sufficient genetic diversity. The specific process is shown in Algorithm 2.

Algorithm 2 Initialization Procedure

1: Input: Job set J, Population size N   2: Output: Initial population $P_0$   3: Sort jobs in J in descending order of total processing times to form list L   4: for each objective $o\in \left\{TT\right(\mathrm{Total}\mathrm{Tardiness}),TEC(\mathrm{Total}\mathrm{Energy}\mathrm{Cos}\mathrm{t}),CTC$ do (Carbon Trading Cost)}   5:       Select the first two jobs from L to generate an initial sequence $\pi$ that minimizes o   6:       for each remaining job in L do   7:             Insert the job into the position in $\pi$ that minimizes o   8:       end for   9:       Add $\pi$ to $P_0$ 10: end for 11: while  $|P_0| < N$  do 12:       Generate a random job permutation $\pi$ 13:       Add $\pi$ to $P_0$ 14: end while

5.2.3. Genetic Operations

Genetic operations are essential for fostering population diversity and guiding the convergence toward optimal solutions. For crossover, three adopted strategies are employed, all illustrated in Figure 2. Partially Mapped Crossover (PMX) swaps subsequences between parent individuals, with a mapping mechanism preserving the relative order of remaining elements to avoid redundancy or omission. Two-Point Order Crossover (2p-OXO) exchanges segments between two randomly designated crossover points, while retaining the integrity of non-crossover regions to balance genetic inheritance and exploratory potential. Uniform Crossover (UX) enables element-wise recombination, with each position inherited from either parent with equal probability to support fine-grained solution refinement.

Figure 2. Schematics of Crossover and Mutation Operations.

For mutation, complementary strategies introduce targeted perturbations. Swap Mutation exchanges values at two randomly selected positions, introducing local adjustments to escape local optima. Insertion Mutation relocates a randomly chosen element to another position within the sequence, modifying structural arrangements to explore novel solution spaces. Inversion Mutation reverses the order of elements in a randomly selected subsequence, further expanding the search scope and enhancing population diversity.

5.3. GWO-Enhanced Exploration Strategy

This module incorporates the vector-based search capabilities of the Grey Wolf Optimizer to alleviate the premature convergence often encountered in standard NSGA-II. By embedding swarm intelligence into the evolutionary process, the algorithm establishes a dynamic equilibrium between genetic variation and directional exploration.

5.3.1. Dual-Path Offspring Generation Mechanism

Conventional NSGA-II relies exclusively on crossover and mutation operators, which may limit the exploration of the solution space in high-dimensional scheduling problems. The QGN algorithm addresses this by implementing a parallel dual-path generation strategy, as illustrated in Figure 3. The process begins with the extraction of the top N individuals from the parent population $P_t$ via tournament selection to construct a high-quality elite set, $P_{elite}$. This elite set functions as the common precursor for two simultaneous evolutionary trajectories. In the first path, the GWO position update mechanism operates on the elites to generate a set of exploratory offspring, denoted as W. Concurrently, the second path applies genetic crossover and mutation operators to the same elite set to produce an evolutionary offspring set, O. These two distinct sets are subsequently amalgamated into a temporary population $T_{tmp}$ with a size of $2N$. To determine the final offspring population for the succeeding generation, the algorithm applies a hierarchical screening criterion: solutions in $T_{tmp}$ are first stratified into Pareto fronts via fast non-dominated sorting, and individuals are recruited layer by layer. When the inclusion of a specific front exceeds the population limit N, individuals within that critical layer are prioritized based on their crowding distance. This dual-path mechanism ensures that the population benefits from both the broad search scope of swarm intelligence and the stochastic diversity of genetic operations.

Figure 3. Flowchart of QGN algorithm.

5.3.2. Adaptive Parameter Control Scheme

The convergence factor a in GWO governs the transition intensity from global exploration to local exploitation. Relying on the static linear decay characteristic of the original GWO restricts the algorithm’s adaptability to the rugged landscapes of energy-efficient scheduling. To enhance robustness, a three-tier control scheme is formulated. A linear decrement strategy serves as the baseline, $a_{base}=2-2g/T$, providing a fundamental guarantee of convergence stability throughout the iterations. Superimposed on this baseline is a dynamic adjustment mechanism driven by the Q-Learning agent, which modulates a in response to the real-time optimization state, such as fluctuations in the Pareto frontier ratio. For instance, detection of a decrease in population diversity triggers an upward adjustment of a to amplify exploration. Furthermore, to prevent parameter divergence, boundary constraints restrict a within the feasible interval $[0,2]$, complemented by a soft constraint mechanism that imposes reward penalties on actions driving a out of bounds. This hierarchical control ensures the algorithm can autonomously shift its search focus according to the specific needs of the optimization phase.

5.3.3. Pareto-Driven Position Update Strategy

Unlike traditional sequential hybrids, the proposed dual-path mechanism allows the population to evolve through two distinct yet interactive trajectories: the genetic path preserves the inheritance of structural traits through crossover and mutation, while the GWO path facilitates directional movement based on social dominance. These paths do not function in isolation; rather, they share a common elite pool and are subject to a unified Pareto-based selection criterion. This synergy ensures that the superior convergence speed of the GWO path and the broad exploratory reach of the NSGA-II path reinforce each other, creating a combined search effect that is theoretically more robust than either method used independently. While the position update logic references the standard mathematical models of GWO defined in Section 3.2, its implementation within the QGN framework exhibits distinct fusion characteristics tailored for multi-objective optimization. Unlike the original GWO, where leadership roles are determined solely by single-objective fitness values, the hierarchy of $\alpha$, $\beta$, and $\delta$ wolves in this hybrid architecture is driven by the Pareto dominance criteria of NSGA-II. Specifically, the leaders are identified not by a scalar cost, but by their rank in the non-dominated sorting and their crowding distance within the $P_{elite}$ set. These high-quality elites, representing the current best approximation of the Pareto front, guide the remaining individuals to update their positions in the continuous vector space. This mechanism effectively transforms the standard GWO update into a directed “super-mutation” operator. By allowing the swarm intelligence component to be guided by Pareto-optimal elites filtered through NSGA-II, the algorithm achieves a synergy where GWO enhances the exploration of the decision space while NSGA-II ensures convergence towards a diverse and optimal trade-off front. This multi-leader guidance mechanism substantiates the choice of GWO over traditional single-trajectory heuristics. By directly mapping the $\alpha ,\beta$, and $\delta$ leadership roles to the Pareto dominance criteria rather than a scalar cost, GWO structurally aligns with the multi-objective nature of the EEHFSP. It allows multiple high-quality elites representing diverse trade-offs to simultaneously pull the population out of local optima, an effect that standard discrete neighborhood operators struggle to replicate.

5.4. Q-Learning-Based Adaptive Control

To navigate the trade-offs inherent in the multi-objective EEHFSP, this module functions as a high-level decision controller within the QGN framework, as illustrated in Figure 1. By establishing a closed loop of “state perception, strategy execution, and feedback evaluation,” the algorithm autonomously governs the optimization trajectory, shifting focus from global exploration to local exploitation based on real-time population dynamics. This adaptive mechanism ensures that the interaction between NSGA-II and GWO is not a simple combination but a strategic synergy optimized to maximize the convergence pressure toward the true Pareto frontier.

5.4.1. State-Action Space Construction

Accurate perception of the optimization environment is the prerequisite for intelligent decision-making. Rather than relying solely on iteration counts, a multidimensional state vector S is constructed to encapsulate the comprehensive status of the population:

$$S=[{\lambda }_g,R_p,D_{avg},F,a]$$

where ${\lambda }_g$ denotes the iteration progress; $R_p$ represents the Pareto frontier ratio indicating convergence level; $D_{avg}$ reflects the solution distribution uniformity; F aggregates the current performance across TT, TEC, and CTC objectives relative to historical optima; and a is the GWO exploration parameter. Based on this perceived state, the agent selects an operation from the discrete action space $A=\{A_1,\dots ,A_9\}$, which serves as a repository of the algorithmic operators defined in the preceding sections. Specifically, the action set is categorized into three strategic groups: genetic crossover operators ($A_1$: PMX, $A_2$: 2p-OXO, $A_3$: UX) to facilitate gene exchange; mutation operators ($A_4$: Swap, $A_5$: Insertion, $A_6$: Inversion) to induce stochastic perturbations; and GWO parameter controls ($A_7$: Increase a, $A_8$: Decrease a, $A_9$: Maintain a) to regulate swarm behavior. This mapping enables the agent to dynamically prioritize operators, such as favoring $A_4$–$A_6$ to escape local optima during stagnation or selecting $A_8$ to enhance convergence precision in later stages. Furthermore, the mathematical formulation of GWO provides an optimal and highly responsive control interface for the reinforcement learning agent. Dynamically modulating a single continuous convergence parameter a offers a streamlined and robust action space.

5.4.2. Multi-Objective Dynamic Reward

To effectively guide the agent toward the Pareto front, a dynamic reward function is formulated that transcends simple objective aggregation. The raw reward $r_{raw}$ is derived from the relative improvements in the three objectives generated by the executed action:

$$r_{raw}={\beta }_{TT}{\delta }_{TT}+{\beta }_{TEC}{\delta }_{TEC}+{\beta }_{CTC}{\delta }_{CTC}.$$

(23)

Here, ${\delta }_j$ represents the relative improvement rate for objective j. To mitigate the bias caused by different magnitudes of these objectives, a normalization process maps the improvements to a unified scale:

$${\delta }_{j,norm}={\displaystyle \frac{{\delta }_j-{\delta }_{j,min}}{{\delta }_{j,max}-{\delta }_{j,min}}}.$$

(24)

Crucially, a dynamic weighting scheme is introduced where weights are inversely proportional to the optimization ranking of each objective:

$${\beta }_j^{dynamic}={\displaystyle \frac{1-ran{k}_j/3}{{\sum }_{k=1}^3(1-ran{k}_k/3)}}.$$

(25)

This mechanism compels the agent to focus on the “weakest” objectives, ensuring balanced optimization across all conflicting goals. Furthermore, to align the learning signal with the evolutionary lifecycle, a stage-dependent modulation is applied to the reward, resulting in the final adjusted reward $r_{adjusted}$:

$$r_{adjusted}=w_{stage}·\left({\beta }_{TT}^{dynamic}{\delta }_{TT,norm}+{\beta }_{TEC}^{dynamic}{\delta }_{TEC,norm}+{\beta }_{CTC}^{dynamic}{\delta }_{CTC,norm}\right),$$

(26)

where $w_{stage}$ acts as a time-variant factor that amplifies rewards for exploration-oriented actions (e.g., GWO adjustments) in the early phases while incentivizing precise objective improvements in the latter phases.

5.4.3. Policy Update and Execution Flow

The core of the control strategy lies in the iterative update of the Q-table, which maps state-action pairs to expected long-term rewards. Utilizing the feedback calculated from the multi-objective reward mechanism, the agent updates its knowledge base following the Temporal Difference error:

$$Q(s,a)\leftarrow Q(s,a)+\alpha [r_{adjusted}+\gamma \underset{a^{\prime }}{\max }Q(s^{\prime },a^{\prime })-Q(s,a)].$$

(27)

The execution flow of the QGN algorithm follows the logic depicted in Figure 3. In each generation, the agent first perceives the current state S and selects an action using an $ϵ$-greedy policy to balance exploration and exploitation. Depending on whether the selected action belongs to the genetic set ($A_1$–$A_6$) or the GWO set ($A_7$–$A_9$), the algorithm branches into either genetic evolution or GWO-based exploration to generate offspring. Following population merging and screening, the adjusted reward $r_{adjusted}$ is computed to update the Q-table. This batch update mechanism stabilizes the learning process by averaging out the stochastic noise inherent in individual evolutionary steps, allowing the algorithm to evolve a robust control policy tailored to the specific landscape of the scheduling instance. The specific process is shown in Algorithm 3.

Algorithm 3 QGN Algorithm

1: Input: Job set, machine parameters, energy consumption/carbon cost data, population size N, maximum iterations T   2: Output: Pareto optimal solution set   3: Set parameters: $\alpha =0.1$, $\gamma =0.9$, $\epsilon =0.1$, initial $a=2$   4: Generate initial population (size N)   5: Initialize Q-table (5-dimensional state × 9 actions, values set to 0)   6: Calculate initial objective values $TT$, $TEC$, $CTC$; set current iteration $g=0$   7: While $g<T$:   8:     Calculate state vector $s=[{\lambda }_g,R_p,D_{\mathrm{avg}},F,a]$   9:     Select parent elite set $Parent$ (size N) 10:     Select action $a_t\in \{A1–A9\}$ (using $\epsilon$-greedy strategy) 11:     If $a_t\in \{A1–A3\}$, generate offspring O via crossover 12:     If $a_t\in \{A4–A6\}$, generate offspring O via mutation 13:     If $a_t\in \{A7–A9\}$, adjust a and generate offspring W via GWO 14:     Normalize O and W, merge into temporary population $Temp$ (size $2N$) 15:     Screen $Temp$ to generate a new population (size N) 16:     Calculate reward $r_t$, obtain new state $s^{\prime }$, and update the Q-table 17:     $g=g+1$ 18: Output non-dominated solutions in the final population

5.5. Power-Down Strategy

To visually demonstrate the proposed energy-saving mechanism, a comparative Gantt chart is presented in Figure 4. In this illustration, the gray dashed blocks represent the standby state where the machine remains idle between two consecutive jobs, whereas the cut-off blocks indicate the machine’s shutdown and subsequent restart phases. By converting long idle periods into shutdown states, the strategy aims to reduce electricity consumption without affecting the completion time of subsequent tasks. The core decision of whether to shut down a machine or keep it in standby mode depends on specific time thresholds. If the idle interval is insufficient to cover the restart cost, the machine remains in standby to ensure efficiency. This dynamic adjustment allows the system to minimize energy waste intelligently.

Figure 4. The effect of the power down strategy. In the Gantt charts, the distinct colors represent different jobs, and the numbers inside the blocks denote their corresponding job IDs.

The inter-arrival time between two consecutive jobs (the q-th job and the $(q+1)$-th job) on a machine $m_{ik}$, denoted as $T{I}_{ik{\sigma }_{ik}\left(q\right){\sigma }_{ik}(q+1)}$, is computed as:

$$T{I}_{ik{\sigma }_{ik}\left(q\right){\sigma }_{ik}(q+1)}=C_{{\sigma }_{ik}(q+1)k}-{\displaystyle \frac{p_{{\sigma }_{ik}(q+1)k}}{v_{ik}}}-C_{{\sigma }_{ik}\left(q\right)k},i\in M_k,k\in S,$$

(28)

where C represents the completion time, p is the processing time, and v is the speed of the machine. The break-even time for shutting down a machine, $T{B}_{ik}$, is defined as:

$$T{B}_{ik}={\displaystyle \frac{t_{reset}{e}_{ik}^r}{e_{ik}^s}},i\in M_k,k\in S,$$

(29)

where $t_{reset}$ is the reset time, $e_{ik}^r$ represents the energy consumption during reset, and $e_{ik}^s$ denotes the standby power.

A machine $m_{ik}$ is powered down if $T{I}_{ik{\sigma }_{ik}\left(q\right){\sigma }_{ik}(q+1)}>T{B}_{ik}$. Otherwise, it remains in standby mode, as the energy consumed during the idle state is lower than the reset cost. Without this strategy, machines remain in a standby state during long gaps between tasks, leading to significant energy waste. In contrast, when the idle time exceeds the break-even threshold, the machine state is successfully converted to shutdown and reset, thereby effectively minimizing the total energy cost.

6. Experiments and Results

6.1. Parameter Configuration

To comprehensively verify the adaptability of the QGN algorithm to different production scenarios, three categories of 9 test cases are designed, covering low-volume and high-mix, medium-volume and standardized, and high-volume and large-scale scenarios. Each test case is denoted in the format “number of jobs–number of stages–number of machines per stage” (abbreviated as n-s-m). For example, “20-5-4” represents a case with 20 jobs, 5 processing stages, and 4 machines in each stage.

Core parameters related to scheduling and energy calculation are configured based on industrial practice and academic standards. The benchmark processing time of job j at stage k follows a uniform distribution $U[4,12]$ with the unit of hour. The processing speed of uniform parallel machines is preset to three discrete levels of 0.7, 1.0, and 1.3, which are dimensionless and relative to the base speed. The speed-power conversion coefficient matches the machine speed, taking dimensionless values of 0.5, 1.0, and 1.6. For the machine power parameters at stage k, the benchmark processing power follows $U[4.5,11]\times {10}^5$ watts; the standby power is calculated as $0.35\times {\lambda }_{ik}\times p{e}_k$ watts; the restart power equals $0.6\times s{e}_{ik}$ watts; and the restart duration is fixed at 1.2 h.

To ensure the realism of the simulation, the experiment adopts the 2024 industrial TOU electricity pricing policy of Zhejiang Province, as shown in Table 2. Environmental and scheduling parameters related to carbon emissions and job delivery are configured as follows: Specifically, the average carbon price is 72 CNY per ton of CO₂, and the carbon emission coefficient per unit of energy is set to 0.21 tons of CO₂ per MWh. The carbon quota is calculated as $1.2\times n\times s$ with the unit of tons of CO₂, where n is the number of jobs, s is the number of stages, and setting the coefficient to 1.2 can create a moderately strict carbon constraint. The job due date ($d_j$) [62] is determined by the formula:

$$d_j=\max \left(0,U\left[P(1-\tau -R/2),P(1-\tau +R/2)\right]\right),$$

(30)

where P denotes the lower bound of the maximum makespan, $\tau =0.3$ acts as the tardiness factor, and $R=0.9$ is the due date factor, $U[a,b]$ denotes a uniform distribution in the interval $[a,b]$.

Table 2. 2024 Industrial TOU Electricity Pricing Policy of Zhejiang Province.

Period Type	Time Range	Electricity Price (CNY/MWh)
Off-peak	22:00–08:00 (next day)	245
Flat	08:00–11:00, 15:00–19:00	610
Peak	11:00–15:00, 19:00–22:00	1020

Universal basic parameters are consistently configured for all algorithms to ensure fair comparison. The population size is set to 120, and the maximum number of iterations is specified as 250. The termination criterion for all algorithms is set to the maximum number of iterations. The crossover probability is 0.8 while the mutation probability is 0.25. For Q-learning parameters, the learning rate $\alpha =0.1$, the discount factor $\gamma =0.9$, and the $\epsilon$-greedy probability $\epsilon =0.1$. The GWO parameter baseline follows a linear decrement rule: $a_{\mathrm{baseline}}=2-2g/T$, where g represents the current iteration and T denotes the total number of iterations.

To comprehensively evaluate the QGN algorithm’s performance, four representative algorithms representing distinct multi-objective optimization paradigms are selected for comparison, with parameters strictly set according to their original studies to ensure benchmark consistency. Specifically, QGN is compared against the classical NSGA-II [17], the improved Jaya [63], the improved MOEA/D [64] and the recent state-of-the-art QVNS algorithm [46].

Three metrics are selected to evaluate algorithm performance from three dimensions: convergence, diversity, and computational efficiency, all of which comply with international academic standards for multi-objective optimization. The Coverage Metric (CM) measures the dominance relationship between two solution sets, with the formula

$$C(A,B)={\displaystyle \frac{\left|\right\{b\in B\mid \exists a\in A:a\succ b\mathrm{or}a=b\left\}\right|}{\left|B\right|}},$$

(31)

its value range is [0, 1], and a value closer to 1 indicates stronger dominance of solution set A over B. The Number of Pareto Solutions (NPS) reflects the quantitative diversity of solutions, referring to the number of non-dominated solutions finally generated by the algorithm, and a larger NPS indicates wider coverage of the objective space. The Spacing Metric (SM) evaluates the uniformity of solution distribution on the Pareto frontier, with the formula

$$SM=\sqrt{{\displaystyle \frac{{\sum }_{i=1}^{\left|A\right|}{(d_i-\overline{d})}^2}{\left|A\right|}}},$$

(32)

where $d_i$ is the normalized distance between solution i and its nearest neighboring solution, $\overline{d}$ is the average of all $d_i$, and a smaller SM indicates more uniform solution distribution.

6.2. Comparative Experiments

The CM results in Table 3 demonstrate that the QGN algorithm achieves strong solution dominance across the test instances. When we compare QGN with classic algorithms like NSGA-II, Improved Jaya, and MOEA/D, the CM values for QGN consistently stay above 0.8. In many cases, this value reaches 1.0. In contrast, the reverse coverage values are almost always 0, which means the solutions from QGN fully dominate the solutions from the other algorithms. Even against the advanced QVNS algorithm, QGN maintains a clear advantage with a hit rate of 12 out of 18. This performance proves that QGN has a significant advantage in convergence. The algorithm can stably approach the true Pareto front, even in large-scale complex instances like 50-5-6.

Table 3. CM Results.

Instance	C(Q,N)		C(N,Q)		C(Q,J)		C(J,Q)		C(Q,M)		C(M,Q)		C(Q,V)		C(V,Q)
	Mean	Std	Mean	Std	Mean	Std	Mean	Std	Mean	Std	Mean	Std	Mean	Std	Mean	Std
10-3-3	0.88	0.12	0.00	0.00	1.00	0.00	0.00	0.00	1.00	0.00	0.00	0.00	0.20	0.04	0.45	0.02
10-3-4	0.89	0.06	0.00	0.00	1.00	0.00	0.00	0.00	1.00	0.00	0.00	0.00	0.24	0.06	0.41	0.04
10-3-6	0.98	0.03	0.00	0.00	0.99	0.01	0.00	0.00	1.00	0.00	0.00	0.00	0.27	0.03	0.23	0.05
10-5-3	0.96	0.07	0.00	0.00	1.00	0.00	0.00	0.00	1.00	0.00	0.00	0.00	0.21	0.05	0.39	0.04
10-5-4	0.86	0.11	0.00	0.00	1.00	0.00	0.00	0.00	1.00	0.00	0.00	0.00	0.14	0.04	0.36	0.03
10-5-6	0.94	0.06	0.00	0.00	1.00	0.00	0.00	0.00	1.00	0.00	0.00	0.00	0.28	0.03	0.30	0.03
20-3-3	0.83	0.18	0.00	0.00	1.00	0.00	0.00	0.00	1.00	0.00	0.00	0.00	0.21	0.06	0.26	0.05
20-3-4	0.95	0.04	0.00	0.00	1.00	0.00	0.00	0.00	1.00	0.00	0.00	0.00	0.17	0.08	0.16	0.04
20-3-6	0.87	0.08	0.00	0.00	1.00	0.00	0.00	0.00	1.00	0.00	0.00	0.00	0.31	0.09	0.19	0.07
20-5-3	0.83	0.13	0.00	0.00	1.00	0.00	0.00	0.00	1.00	0.00	0.00	0.00	0.29	0.08	0.20	0.03
20-5-4	0.91	0.09	0.00	0.00	1.00	0.00	0.00	0.00	1.00	0.00	0.00	0.00	0.27	0.06	0.16	0.08
20-5-6	0.85	0.11	0.00	0.00	1.00	0.00	0.00	0.00	1.00	0.00	0.00	0.00	0.36	0.07	0.23	0.06
50-3-3	0.93	0.05	0.00	0.00	1.00	0.00	0.00	0.00	1.00	0.00	0.00	0.00	0.56	0.04	0.20	0.06
50-3-4	0.87	0.03	0.00	0.00	1.00	0.00	0.00	0.00	1.00	0.00	0.00	0.00	0.63	0.08	0.17	0.03
50-3-6	0.91	0.06	0.00	0.00	1.00	0.00	0.00	0.00	1.00	0.00	0.00	0.00	0.59	0.05	0.09	0.04
50-5-3	0.82	0.14	0.00	0.00	1.00	0.00	0.00	0.00	1.00	0.00	0.00	0.00	0.66	0.06	0.11	0.03
50-5-4	0.93	0.06	0.00	0.00	1.00	0.00	0.00	0.00	1.00	0.00	0.00	0.00	0.63	0.08	0.10	0.02
50-5-6	0.86	0.04	0.00	0.00	1.00	0.00	0.00	0.00	1.00	0.00	0.00	0.00	0.61	0.06	0.08	0.02
Hit rate	18/18		0/18		18/18		0/18		18/18		0/18		12/18		6/18

Table 4 presents the NPS results, which further confirms the advantages of QGN. The algorithm obtains the highest number of non-dominated solutions in 12 out of 18 instances, and its average NPS is significantly higher than that of the comparative algorithms. In small and medium-scale instances, the NPS of QGN is often much larger than the NPS of the Jaya algorithm. When the problem scale expands to 50 jobs and 5 stages, QGN continues to maintain a rich set of solutions, while the NPS of other algorithms declines noticeably. For example, in instance 50-5-6, QGN finds an average of 46 solutions, significantly surpassing the 24 solutions found by QVNS. These results indicate that the dual-path search mechanism in QGN effectively expands the coverage of the solution space and avoids the loss of diversity caused by premature convergence.

Table 4. NPS Results.

Instance	NSGA-II		Jaya		MOEA/D		QVNS		QGN
	Mean	Std	Mean	Std	Mean	Std	Mean	Std	Mean	Std
10-3-3	28	7	7	2	12	4	37	7	39	6
10-3-4	33	9	6	2	9	3	41	9	43	8
10-3-6	58	8	22	5	15	5	67	6	52	11
10-5-3	35	12	9	3	8	3	50	4	58	7
10-5-4	34	14	8	4	7	2	68	7	59	9
10-5-6	48	11	9	3	18	6	93	4	89	8
20-3-3	25	6	5	2	12	4	43	6	36	4
20-3-4	35	8	11	3	13	3	34	8	52	6
20-3-6	24	9	6	2	14	4	53	7	35	5
20-5-3	27	6	5	3	6	2	38	9	61	3
20-5-4	24	7	5	2	8	3	40	5	41	5
20-5-6	31	10	8	4	6	2	37	7	48	7
50-3-3	24	11	8	3	7	2	24	3	42	5
50-3-4	22	13	6	3	8	3	29	7	45	6
50-3-6	18	6	5	2	4	2	28	8	33	4
50-5-3	45	23	7	4	7	3	25	4	38	14
50-5-4	29	14	10	4	12	4	26	9	54	8
50-5-6	23	8	8	3	6	2	24	8	46	5
Hit rate	1/18		0/18		0/18		5/18		12/18

The best results are highlighted in bold.

Regarding the distribution of solutions, the SM results in Table 5 shows that the Pareto solutions from QGN are distributed more uniformly. The algorithm achieves the best hit rate of 11 out of 18, and its average SM value is the lowest in most instances. QGN also maintains a smaller standard deviation, which indicates low dispersion. In contrast, the SM values for the Jaya algorithm and MOEA/D are generally higher, which suggests that their solutions tend to cluster together. Notably, while the state-of-the-art QVNS achieves excellent convergence, its SM values are often slightly higher than QGN’s in complex instances. This phenomenon occurs because the GVNS operator intensifies search around specific local optima, causing solutions to cluster tightly together rather than spreading uniformly. In contrast, QGN utilizes the GWO-based vector updates to guide the population toward diverse regions while relying on the crowding distance mechanism of NSGA-II to maintain spacing. This dual mechanism ensures that QGN generates the most structurally balanced Pareto fronts.

Table 5. SM Results.

Instance	NSGA-II		Jaya		MOEA/D		QVNS		QGN
	Mean	Std	Mean	Std	Mean	Std	Mean	Std	Mean	Std
10-3-3	1.45 × 10−1	4.25 × 10−2	3.88 × 10−1	5.61 × 10−2	3.12 × 10−1	1.05 × 10−1	6.65 × 10−2	2.13 × 10−2	6.92 × 10−2	2.15 × 10−2
10-3-4	1.88 × 10−1	2.12 × 10−2	3.19 × 10−1	2.19 × 10−1	2.45 × 10−1	1.85 × 10−1	6.94 × 10−2	3.67 × 10−2	5.25 × 10−2	1.25 × 10−2
10-3-6	4.85 × 10−2	1.45 × 10−2	2.49 × 10−1	8.05 × 10−2	8.95 × 10−2	5.15 × 10−2	5.87 × 10−2	2.43 × 10−2	5.12 × 10−2	9.45 × 10−3
10-5-3	2.15 × 10−1	8.45 × 10−2	2.66 × 10−1	1.66 × 10−1	1.95 × 10−1	9.25 × 10−2	5.36 × 10−2	1.63 × 10−2	6.35 × 10−2	1.65 × 10−2
10-5-4	1.42 × 10−1	4.55 × 10−2	1.95 × 10−1	3.85 × 10−2	3.15 × 10−1	6.25 × 10−2	6.54 × 10−2	3.69 × 10−2	7.05 × 10−2	2.25 × 10−2
10-5-6	6.12 × 10−2	2.15 × 10−2	2.62 × 10−1	8.58 × 10−2	2.95 × 10−1	1.15 × 10−1	5.37 × 10−2	6.23 × 10−2	4.42 × 10−2	1.05 × 10−2
20-3-3	2.45 × 10−1	1.85 × 10−1	1.85 × 10−1	9.45 × 10−2	1.65 × 10−1	5.65 × 10−2	9.42 × 10−2	5.07 × 10−2	5.85 × 10−2	3.35 × 10−2
20-3-4	1.92 × 10−1	6.45 × 10−2	3.15 × 10−1	1.65 × 10−1	1.88 × 10−1	8.45 × 10−2	7.02 × 10−2	3.37 × 10−2	4.95 × 10−2	1.15 × 10−2
20-3-6	1.75 × 10−1	1.15 × 10−1	3.45 × 10−1	1.95 × 10−1	2.05 × 10−1	1.25 × 10−1	9.78 × 10−2	4.72 × 10−2	5.32 × 10−2	4.45 × 10−2
20-5-3	8.15 × 10−2	2.45 × 10−2	2.95 × 10−1	2.05 × 10−1	2.45 × 10−1	2.35 × 10−1	7.98 × 10−2	3.05 × 10−2	8.45 × 10−2	2.15 × 10−2
20-5-4	1.55 × 10−1	7.15 × 10−2	5.92 × 10−2	1.18 × 10−1	1.35 × 10−1	4.25 × 10−2	8.31 × 10−2	4.23 × 10−2	6.15 × 10−2	1.95 × 10−2
20-5-6	1.15 × 10−1	3.15 × 10−2	5.12 × 10−1	6.45 × 10−2	1.45 × 10−1	9.15 × 10−2	6.47 × 10−2	5.41 × 10−2	4.85 × 10−2	2.12 × 10−2
50-3-3	1.65 × 10−1	1.05 × 10−1	3.85 × 10−1	1.65 × 10−1	6.85 × 10−2	6.45 × 10−2	7.63 × 10−2	2.61 × 10−2	7.05 × 10−2	1.75 × 10−2
50-3-4	1.45 × 10−1	4.85 × 10−2	3.55 × 10−1	1.45 × 10−1	9.45 × 10−2	2.15 × 10−2	8.47 × 10−2	2.58 × 10−2	5.35 × 10−2	1.35 × 10−2
50-3-6	1.25 × 10−1	5.15 × 10−2	4.95 × 10−1	8.45 × 10−2	1.15 × 10−1	1.45 × 10−1	7.94 × 10−2	1.83 × 10−2	4.65 × 10−2	9.85 × 10−3
50-5-3	1.15 × 10−1	4.65 × 10−2	3.28 × 10−1	1.72 × 10−2	1.85 × 10−1	1.15 × 10−1	7.57 × 10−2	2.52 × 10−2	6.15 × 10−2	1.45 × 10−2
50-5-4	1.75 × 10−1	6.15 × 10−2	2.87 × 10−1	6.72 × 10−2	1.95 × 10−1	8.45 × 10−2	8.27 × 10−2	4.76 × 10−2	4.75 × 10−2	1.15 × 10−2
50-5-6	1.35 × 10−1	3.15 × 10−2	3.66 × 10−1	2.16 × 10−1	1.65 × 10−1	6.15 × 10−2	8.76 × 10−2	2.44 × 10−2	5.15 × 10−2	1.25 × 10−2
Hit rate	1/18		1/18		1/18		4/18		11/18

The best results are highlighted in bold.

The statistical stability of the algorithm is visually corroborated by the SM boxplot in Figure 5. The boxplot for QGN is positioned at the lowest point on the vertical axis, which confirms its superior average uniformity. More importantly, the height of the box for QGN is the narrowest among all the algorithms. This narrow shape indicates minimal variance across independent runs. On the other hand, Improved Jaya and MOEA/D show larger medians and wider dispersion, which reflects instability in their optimization processes. The tight distribution of QGN confirms that the adaptive control mechanism successfully reduces stochastic volatility and ensures the consistent generation of high-quality scheduling schemes.

Figure 5. The boxplot of SM results.

6.3. Ablation Experiments

To isolate the specific contributions of Q-learning, the GWO, and the Dynamic Reward Mechanism, we evaluated three distinct variants against the full QGN framework across diverse production scales. These variants included a configuration without Reinforcement Learning (Q-1), one excluding GWO (Q-2), and another substituting dynamic rewards with fixed weights (Q-3). The dominance analysis in Table 6 demonstrates that the complete QGN algorithm consistently outperforms these simplified versions. The Q-2 variant suffers the most significant degradation, with QGN dominating approximately 91% of its solutions in the large-scale instance. This sharp decline confirms that GWO’s vector-based search is indispensable for escaping local optima. Similarly, the Q-1 variant trails behind with a lower coverage metric, proving that the RL-driven adaptive strategy offers superior search efficiency compared to random operator selection.

Table 6. CM Results of Ablation Experiments.

Case	C(Q,Q-1)		C(Q,Q-2)		C(Q,Q-3)
	Mean	Std	Mean	Std	Mean	Std
Case 1	0.65	0.12	0.76	0.09	0.25	0.08
Case 2	0.78	0.10	0.83	0.07	0.32	0.10
Case 3	0.88	0.06	0.91	0.03	0.45	0.11

Further evidence from Table 7 and Table 8 highlights the impact of these modules on solution diversity and uniformity. QGN identifies the highest number of non-dominated solutions, finding 54 in the largest test case, whereas the GWO-free Q-2 variant drops to 32 due to premature convergence. Meanwhile, the Q-3 variant specifically exposes the necessity of the balancing mechanism. The reliance on fixed weights causes the Spacing Metric to deteriorate significantly, as the population clusters around specific objectives without the compensatory effect of dynamic rewards. These findings collectively establish that GWO secures the baseline solution quality, the RL agent maximizes search efficiency, and the Dynamic Reward mechanism ensures a balanced trade-off among conflicting objectives.

Table 7. NPS Results of Ablation Experiments.

Case	Q		Q-1		Q-2		Q-3
	Mean	Std	Mean	Std	Mean	Std	Mean	Std
Case 1	59	9	40	9	36	5	48	4
Case 2	41	5	28	6	26	4	33	3
Case 3	54	8	37	5	32	8	36	7

Table 8. SM Results of Ablation Experiments.

Case	Q		Q-1		Q-2		Q-3
	Mean	Std	Mean	Std	Mean	Std	Mean	Std
Case 1	7.05 × 10−2	2.25 × 10−2	1.35 × 10−1	4.15 × 10−2	1.62 × 10−1	5.25 × 10−2	1.08 × 10−1	3.45 × 10−2
Case 2	6.15 × 10−2	1.95 × 10−2	1.25 × 10−1	4.65 × 10−2	1.48 × 10−1	6.15 × 10−2	9.45 × 10−2	2.85 × 10−2
Case 3	4.75 × 10−2	1.15 × 10−2	1.05 × 10−1	3.15 × 10−2	1.28 × 10−1	4.85 × 10−2	7.85 × 10−2	2.45 × 10−2

7. Conclusions

This study addresses the Energy-Efficient Hybrid Flow Shop Scheduling Problem, aiming to optimize the conflicting objectives of total tardiness, energy costs, and carbon trading costs simultaneously. To effectively navigate the complex trade-offs between global exploration and local exploitation, we developed a hybrid metaheuristic algorithm termed QGN. This approach synergizes the global exploration capability of the Grey Wolf Optimizer with the evolutionary structure of NSGA-II, while embedding a Q-learning module to function as a high-level decision controller. By establishing a closed-loop mechanism of state perception and feedback, the algorithm adaptively regulates genetic operators and search parameters in response to real-time population dynamics, thereby overcoming the rigidity inherent in static evolutionary strategies.

Empirical validation across diverse production scales confirms that QGN significantly outperforms standard baselines, including NSGA-II, Jaya, and MOEA/D. Moreover, the results demonstrate that QGN maintains a clear competitive advantage over the advanced QVNS algorithm. Specifically, the experimental results demonstrate that the proposed algorithm achieves superior solution dominance, covering a broader region of the objective space compared to comparative methods. The algorithm also maintains a more uniform distribution along the Pareto front, effectively mitigating the clustering issues often observed in high-dimensional optimization. Furthermore, ablation studies verify that the integration of adaptive learning and swarm-based exploration is critical for sustaining population diversity and preventing premature convergence, particularly in complex, large-scale scheduling scenarios.

While the QGN algorithm demonstrates superior performance in balancing convergence and diversity for the EEHFSP, certain limitations within the current framework merit discussion. First, regarding computational complexity, the integration of the Q-learning agent and the dual-path search mechanism introduces a marginal increase in computational overhead compared to simple heuristics. Although this additional cost is justified by the significant improvement in solution quality, it implies that the algorithm may require optimization for ultra-large-scale real-time scheduling scenarios. Second, the current model assumes a deterministic production environment. While the algorithm effectively manages the complexities of heterogeneous energy consumption, it does not yet account for dynamic stochastic events such as sudden machine breakdowns or urgent order insertions during the execution phase. These aspects represent promising directions for future research to further enhance the algorithm’s robustness in uncertain industrial settings.

Future research will extend this framework to dynamic manufacturing environments characterized by uncertainties, including stochastic machine failures and urgent order insertions. Additionally, investigations into parallel computing strategies or lightweight network architectures are planned to further enhance computational efficiency for ultra-large-scale scheduling scenarios.