A Bi-Level Intelligent Control Framework Integrating Deep Reinforcement Learning and Bayesian Optimization for Multi-Objective Adaptive Scheduling in Opto-Mechanical Automated Manufacturing

Abstract

The opto-mechanical automated manufacturing process, characterized by stringent process constraints, dynamic disturbances, and conflicting optimization objectives, presents significant control challenges for traditional scheduling and control approaches. We formulate the scheduling problem within a closed-loop control paradigm and propose a novel bi-level intelligent control framework integrating Deep Reinforcement Learning (DRL) and Bayesian Optimization (BO). The core of our approach is a bi-level intelligent control framework. An inner DRL agent acts as an adaptive controller, generating control actions (scheduling decisions) by perceiving the system state and learning a near-optimal policy through a carefully designed reward function, while an outer BO loop automatically tunes the DRL’s hyperparameters and reward weights for superior performance. This synergistic BO-DRL mechanism facilitates intelligent and adaptive decision-making. The proposed method is extensively evaluated against standard meta-heuristics, including Genetic Algorithm (GA) and Particle Swarm Optimization (PSO), on a complex 20-jobs × 20-machines flexible job shop scheduling benchmark specific to opto-mechanical automated manufacturing. The experimental results demonstrate that our BO-DRL algorithm significantly outperforms these benchmarks, achieving reductions in makespan of 13.37% and 25.51% compared to GA and PSO, respectively, alongside higher machine utilization and better on-time delivery. Furthermore, the algorithm exhibits enhanced convergence speed, superior robustness under dynamic disruptions (e.g., machine failures, urgent orders), and excellent scalability to larger problem instances. This study confirms that integrating DRL’s perceptual decision-making capability with BO’s efficient parameter optimization yields a powerful and effective solution for intelligent scheduling in high-precision manufacturing environments.

1. Introduction

The advent of Industry 4.0 and the smart manufacturing paradigm has fundamentally transformed modern production systems, creating a pressing need for highly efficient, adaptive, and intelligent scheduling solutions [1,2]. As a critical decision-making layer in manufacturing execution, Advanced Planning and Scheduling (APS) systems are pivotal for optimizing resource utilization, ensuring on-time delivery, and enhancing overall operational efficiency [3,4]. This is particularly critical in high-precision domains such as opto-mechanical automated manufacturing, where scheduling accuracy directly correlates with final product quality and production line stability [5,6].

The opto-mechanical automated manufacturing process represents a quintessential and highly constrained instance of the Flexible Job Shop Scheduling Problem (FJSP) [7,8]. This domain is characterized by stringent process-specific constraints (e.g., cleanliness grades, thermal stability, vibration isolation) [9], frequent dynamic disruptions (e.g., machine breakdowns, high-priority urgent orders) [10], and inherently conflicting optimization objectives spanning efficiency, utilization, timeliness, and quality [11,12]. The combinatorial complexity of such problems, with solution spaces regularly exceeding 10²⁰⁰ for practical scenarios, firmly classifies them as strongly NP-hard, rendering traditional optimization methodologies increasingly inadequate [13,14].

Traditional approaches to production scheduling have evolved from exact mathematical programming to heuristic and meta-heuristic algorithms [15,16]. While exact methods guarantee optimality, their computational cost becomes prohibitive for real-world problem sizes [17]. Meta-heuristics like the Genetic Algorithm (GA) [18], Particle Swarm Optimization (PSO) [19], and their advanced variants have demonstrated considerable success in navigating complex solution landscapes [20,21]. However, these population-based search strategies often exhibit significant limitations in modern manufacturing contexts, including slow convergence in high-dimensional spaces, a pronounced tendency to become trapped in local optima, and a fundamental lack of built-in responsiveness to dynamic environmental changes [22,23].

The rise of Deep Reinforcement Learning (DRL) has introduced a transformative approach to solving sequential decision-making problems [24,25]. By integrating the representational power of deep neural networks with the decision-making framework of reinforcement learning, DRL agents can learn optimal policies through direct interaction with their environment [26,27]. This capability has led to groundbreaking applications in various scheduling domains, where DRL models learn to map complex system states directly to effective scheduling actions [28,29]. For instance, ref. [30] applied DRL to dynamic job-shop scheduling, demonstrating its potential in environments subject to uncertainty. Nevertheless, the practical application of DRL in manufacturing is hampered by several persistent challenges, including sparse reward signals, the credit assignment problem over extended time horizons, and acute sensitivity to the settings of numerous hyperparameters [31,32].

Concurrently, Bayesian Optimization (BO) has gained prominence as a sample-efficient methodology for optimizing black-box functions, making it particularly well-suited for hyperparameter tuning of complex models like DRL agents [33,34]. BO constructs a probabilistic surrogate model to approximate the objective function and employs an acquisition function to intelligently guide the search process [35,36]. Research such as [37] has explored BO for scheduling. While BO excels in parameter optimization tasks, it lacks the sophisticated, end-to-end perceptual decision-making capabilities inherent to DRL [38,39].

Emerging research has begun to explore the synergistic potential of combining DRL and BO [40,41]. Initial studies have typically focused on using BO as an offline tool to tune DRL hyperparameters [42] or have addressed multi-objective aspects separately [43]. However, a significant gap remains in the development of a tightly coupled, bi-level intelligent control framework where DRL and BO operate in a synergistic, closed-loop manner, continuously adapting to complex, dynamic environments like opto-mechanical automated manufacturing. From a control theory perspective, the scheduling problem in such a manufacturing system can be formulated as the task of synthesizing a high-level, discrete control policy for a Discrete-Event Dynamic System (DEDS), where the control actions are the dispatching and sequencing decisions. A schematic overview of these conventional scheduling approaches and their inherent limitations is presented in Figure 1.

To bridge this research gap, this paper proposes a novel multi-objective adaptive APS algorithm that deeply integrates Deep Reinforcement Learning and Bayesian Optimization within a unified bi-level architecture. Our framework features an inner DRL agent that functions as a real-time scheduler, making decisions based on a comprehensively designed state space, while an outer BO loop acts as a meta-optimizer, automatically and efficiently tuning the DRL’s critical hyperparameters and reward function weights. This creates a self-improving system that significantly enhances scheduling intelligence, adaptability, and performance in dynamic high-precision manufacturing settings.

The main contributions of this work are fourfold:

• A novel bi-level intelligent control framework: We propose an integrated BO-DRL architecture that enables synergistic cooperation between perceptual decision-making and efficient parameter optimization, facilitating continuous system self-improvement.
• Domain-Specific Modeling and Benchmarking: We formalize a complex, large-scale scheduling benchmark for opto-mechanical automated manufacturing, incorporating realistic constraints to provide a rigorous testbed for advanced scheduling algorithms.
• Comprehensive Multi-Objective DRL Design: We develop a tailored DRL model featuring a graph neural network-based state encoder, a hierarchical action space, and a shaped reward function that dynamically balances competing objectives.

Extensive Empirical Validation: We conduct thorough experiments demonstrating our algorithm’s superior performance over established meta-heuristics and alternative approaches in terms of solution quality, convergence speed, robustness, and scalability.

The remainder of this paper is structured as follows: Section 2 details the problem background and description. Section 3 presents the mathematical formulation and complexity analysis. Section 4 elaborates on the proposed BO-DRL algorithm. Section 5 discusses experimental results and analysis. Finally, Section 8 concludes the paper and suggests future research directions.

2. Problem Background and Description

Opto-mechanical automated manufacturing constitutes a critical stage in precision manufacturing, and its scheduling poses significant theoretical and practical research challenges. This study addresses the problem of governing a large-scale manufacturing system through scheduling, which can be modeled as a Discrete-Event Dynamic System (DEDS). The objective is to synthesize a control policy that optimizes system performance. This example not only has the general characteristics of the standard FJSP, but also incorporates special process constraints in the field of opto-mechanical automated manufacturing and adjustment, providing a strict test benchmark for evaluating advanced scheduling algorithms. Compared with ordinary scheduling examples, the uniqueness of this example is reflected in three aspects: first, machine resources are highly heterogeneous and specialized, and each piece of equipment corresponds to specific process requirements; second, the job process contains complex sequence constraints and quality dependencies; finally, the production environment needs to consider dynamic disturbances and process stability requirements. These characteristics make this example closer to the actual manufacturing scenario and provide higher validity for algorithm performance evaluation.

3. Mathematical Model Analysis

This section presents the mathematical formulation and complexity analysis of the multi-objective adaptive scheduling problem for opto-mechanical manufacturing. We begin by defining the optimization objectives and the system constraints. Subsequently, we analyze the combinatorial complexity of the problem and provide a detailed description of the heterogeneous machine resources and production workflow that characterize the benchmark scenario.

3.1. Optimization Objectives and Constraints

The scheduling problem is formulated as a multi-objective optimization. The primary and composite objectives are defined, along with the key constraints that ensure solution feasibility and model the specific process rules of the domain. To enhance clarity, the key variables used throughout the mathematical model are summarized in

Table 1. Nomenclature of key variables in the mathematical model.

Symbol	Description
$i$, $l$	Index for jobs; used to denote distinct jobs when formulating pairwise constraints (e.g., $i\ne l$ ).
$j$, $m$	Index for operations within a job; used to denote distinct operations (e.g., operation $j$ of job $i$ vs. operation $m$ of job $l$).
$k$	Index for machines
$t$	Index for time (iteration in PSO/DRL context)
$C_{max}$	Makespan (maximum completion time)
$C_i$	Completion time of job $i$
$U$	Average machine utilization
$T_{max}$	Maximum tardiness
$Q$	Quality performance index
$s_{ij}$	Start time of operation $j$ of job $i$
$p_{ijk}$	Processing time of operation $O_{ij}$ on machine $k$
$y_{ijlmk}$	Binary variable; equals 1 if operations $O_{ij}$ and $O_{lm}$ are both processed on machine $k$, and $O_{ij}$ precedes $O_{lm}$
$x_{ijk}$	Binary variable; equals 1 if operation $O_{ij}$ is assigned to machine $k$
$d_i$	Due date of job $i$
${\mathcal{M}}_{\mathcal{i}\mathcal{j}}$	Set of machines capable of processing operation $O_{ij}$
$M_{\infty }$	A sufficiently large positive number
$N_j$	Total number of jobs
$M$	Total number of machines
$N_{\mathrm{tardy}}$	Number of tardy jobs
${\mathbb{1}}_{\cdot }$	Indicator function (returns 1 if condition is true, else 0)
$C_{max}^{\mathrm{static}}$	Makespan under the static baseline schedule
$C_{max}^{\mathrm{dynamic}}$	Makespan under dynamic disruptions

.

The primary objective is to minimize the makespan:

$$\mathrm{Minimize} C_{max}=\max \left\{C_i|i=1,2,\dots ,20\right\}$$

(1)

Additionally, we consider the following multi-objective optimization:

$$\mathrm{Minimize} F=\left[C_{max},U,T_{max},Q\right]$$

(2)

Among them, $U$ is the average machine utilization, $T_{max}$ is the maximum delay time, and $Q$ is the quality index.

In order to comprehensively evaluate the performance of the scheduling algorithm, a multi-dimensional evaluation index system is adopted, as detailed in

Table 2. Algorithm performance evaluation indicators.

Indicator Type	Specific Indicators	Calculation Formula
efficiency index	makespan	$C_{max}$
efficiency index	mean flow time	${\displaystyle \frac{1}{N_j}}\sum C_i$
Resource Utilization	Machine utilization	${\displaystyle \frac{1}{M}}{\displaystyle \sum _{k=1}^M}{\displaystyle \frac{\sum p_{ijk}}{C_{max}}}$
Timeliness Indicators	Number of tardy jobs	$N_{\mathit{tardy}}={\displaystyle \sum _{i=1}^{N_j}}{\mathbb{1}}_{\{C_i>d_i\}}$
Robustness Index	Disturbance recovery capability	${\displaystyle \frac{C_{max}^{\mathit{dynamic}}}{C_{max}^{\mathit{static}}}}$

. This system encompasses metrics for time efficiency (e.g., makespan, mean flow time), resource utilization (machine utilization), delivery timeliness (number of tardy jobs), and system robustness (disturbance recovery capability). Together, these indicators provide a balanced assessment of scheduling quality across the key operational objectives in manufacturing.

Constraints include:

$$s_{ij}+p_{ijk}\le s_{i\left(j+1\right)}\hspace{1em}\forall i,j,k$$

(3)

$$s_{ij}+p_{ijk}\le s_{lm}+M_{\mathrm{\infty }}\left(1-y_{ijlmk}\right)\forall \left(i,j\right)\ne \left(l,m\right),k$$

(4)

$$y_{ijlmk}\in \left\{0,1\right\}\forall i,j,l,m,k$$

(5)

$$\sum _{k\in \mathcal{M}ij}{x}_{ijk}=1\hspace{1em}\forall i,j$$

(6)

$$x_{ijk}\in \left\{0,1\right\}\forall i,j,k$$

(7)

The mathematical model established above formally defines the multi-objective scheduling problem for opto-mechanical manufacturing. This formulation encapsulates the system’s key constraints, including operation precedence, machine eligibility, and disjunctive scheduling relations. Together with the variables summarized in Table 1, it provides a precise and complete representation of the decision space and performance objectives. The inherent complexity of this model, which stems from the interaction of binary decision variables, sequencing constraints, and competing objectives, directly motivates the need for the advanced intelligent scheduling approach developed in the subsequent sections.

3.2. Problem Complexity Analysis

This calculation example contains 20 special equipment, divided into 5 functional categories, with their specific descriptions and constraints summarized in

Table 3. Machine resource classification and function description.

Category	Machine IDs	Function Description	Special Constraints
Cleaning	M6, M10, M12	Optical component cleaning and dust removal	Cleanliness level requirements
Precision coating	M4, M11	Optical surface coating treatment	Temperature and humidity control
High-precision assembly	M3, M8, M9, M13, M18	Opto-mechanical integrated assembly	Vibration isolation
Calibration test	M1, M2, M5, M15, M16, M20	Optical performance calibration and testing	Constant temperature environment
special handling	M7, M14, M17, M19	Special process treatment	Dedicated equipment

. The table not only lists the categories but also provides the specific machine identifiers (e.g., M6, M10, M12 for Cleaning), creating a precise mapping between each machine and its functional role. This explicit mapping is crucial for traceability throughout the analysis and directly defines the heterogeneous resource pool with its specialized constraints that the scheduling algorithm must navigate.

The production workflow involving these heterogeneous machines is illustrated in Figure 2, showing the sequential dependencies and specialized processing requirements.

To concretely illustrate the job-specific sequencing and machine flexibility,

Table 4. Operation sequence and machine flexibility for a representative job (Job 1).

Operation of Job 1	Op1	Op2	Op3	Op4	Op5
Equipment (working hours)	M2 (9.5 h), M6 (2.4 h), M10 (8 h)	M9 (8.9 h)	M1 (6.6 h), M17 (5.7 h), M19 (5.2 h)	M6 (0.8 h)	M7 (3.8 h), M19 (7.8 h), M1 (2 h)
operation of job 1	Op6	Op7	Op8	Op9	Op10
Equipment (working hours)	M15 (6.4 h), M20 (1.7 h), M5 (4.7 h)	M15 (9.5 h), M20 (4.3 h), M5 (5.4 h)	M17 (6.3 h), M5 (7.5 h)	M13 (3.4 h), M9 (5.1 h)	M15 (0.8 h), M16 (4.1 h)

Note: This table details the process plan for Job 1 (from the 20 × 20 benchmark). Its 10 operations (Op1 to Op10) must be processed in sequence. Critically, the machines listed under each operation (e.g., M2, M6, M10 for Op1) are parallel, alternative resources for that operation, from which the scheduler selects one. Different jobs have distinct sequences and eligible machine sets.

details the complete process plan for a representative job (Job 1) from the 20 × 20 benchmark instance. Job 1 must follow a fixed internal sequence of operations, each of which can be processed on one of several alternative machines with varying times. This combination of precedence constraints and machine flexibility encapsulates the core scheduling challenge addressed in this work.

The combinatorial complexity of this example can be formally expressed as:

$$Solution space size=\prod _{i=1}^{N_j}\left(O_i!\times \prod _{j=1}^{O_i}{M}_{ij}\right)$$

(8)

where $\mathcal{O}i$ represents the operation set of job $i$, $\left|\mathcal{O}i\right|\in \left[7,10\right]$, and $\mathcal{M}ij$ represents the set of available machines, $\left|\mathcal{M}ij\right|\in \left[1,4\right]$. After calculation, the solution space scale exceeds ${10}^{200}$, which is a strongly NP-hard problem.

4. Problem Difficulties and Algorithm Challenges

Compared with the standard FJSP example, the main difficulties in this example are reflected in three aspects: First, the opto-mechanical automated manufacturing and adjustment process includes strict cleanliness requirements, thermal stability constraints, and calibration dependencies. These constraints form a nonlinear feasible region and significantly increase the search difficulty. The second is the essential conflict between the quality index $Q$ and the time index $C_{max}$. High-precision assembly requires longer stabilization time and fine adjustments, while efficiency goals require fast turnaround. This multi-objective Pareto front search is far more complex than single-objective optimization. Third, opto-mechanical automated manufacturing and adjustment are extremely sensitive to environmental changes. Small temperature fluctuations $\mathsf{\Delta }T$ or vibration interference may require rescheduling. This dynamic nature makes static scheduling solutions have limited effectiveness, requiring algorithms with online adjustment capabilities.

Traditional metaheuristic algorithms face significant challenges when processing this example:

• Genetic Algorithm (GA) suffers from difficulties in maintaining feasibility under complex constraints through its crossover and mutation operations, especially for sequence-dependent calibration constraints $\mathcal{G}\left(O_i\right)$ Its population-based search mechanism is prone to becoming trapped in local optima in such high-dimensional spaces.
• Particle Swarm Optimization (PSO) is inherently mismatched with discrete scheduling problems due to its continuous optimization characteristics. Although encoding transformations enable its application to scheduling, the physical meaning of the position update formula $v_{id}^{t+1}=w{v}_{id}^t+c_1{r}_1\left(p_{id}-x_{id}^t\right)+c_2{r}_2\left(p_{gd}-x_{id}^t\right)$ becomes ambiguous in discrete space. In this formula, $v_{id}^t$ and $x_{id}^t$ represent the velocity and position of particle $i$ in dimension $d$ at iteration $t$, respectively; $p_{id}$ is the particle’s personal best position (cognitive component); $p_{gd}$ is the swarm’s global best position (social component); $w$ is the inertia weight; $c_1$ and $c_2$ are the cognitive and social acceleration coefficients; and $r_1$, $r_2$ are random numbers uniformly distributed in $\left[0,1\right]$. The continuous nature of this update mechanism makes the search susceptible to local optima in discrete scheduling spaces and significantly reduces the probability of finding the global optimum.
• Standard Reinforcement Learning (RL) faces critical challenges of sparse rewards and difficult credit assignment. During long sequential decision-making processes, the backpropagation of the final performance metric $C_{max}$ to intermediate decision steps results in low learning efficiency. Additionally, the enormous dimensionality of the state space $\mathcal{S}$ (formally defined in Section 5.1.1) makes training function approximators particularly challenging.

5. Multi-Objective Adaptive APS Algorithm

This study proposes an adaptive hybrid intelligent control framework that integrates Deep Reinforcement Learning (DRL), Bayesian Optimization (BO), and Multi-Objective Evolutionary Algorithm (MOEA). The inner layer employs a DRL agent as a rapid-response decision-maker, responsible for real-time scheduling under given hyperparameters, while an outer BO loop acts as a meta-controller, tasked with automatically tuning the inner controller’s (DRL agent’s) hyperparameters and reward function weights. These two components work in synergy, forming a bi-level optimization system that enhances scheduling effectiveness, response speed, and decision-making intelligence of the APS system in complex and dynamic environments. The overall architecture of the proposed bi-level BO-DRL framework is depicted in Figure 3, illustrating the synergistic interaction between the inner DRL agent and outer BO optimizer.

5.1. Deep Reinforcement Learning Decision Model Design

5.1.1. State Space Design ($s_t\in \mathcal{S}$)

The state $s_t$ at decision time $t$ is an element of the state space $\mathcal{S}$. It is designed to comprehensively capture both static and dynamic information of the entire production line at time $t$. We design a multi-dimensional feature vector:

$$s_t=\left[s_t^{\mathrm{machine}},s_t^{\mathrm{job}},s_t^{\mathrm{global}}\right]$$

(9)

• Machine State Vector ($s_t^{\mathit{machine}}$): For each machine $M_j$, it includes: current machine status, processed time of the current operation, number of operations in the current queue, and utilization rate within the recent time window.
• Job State Vector ($s_t^{\mathit{job}}$): For each job $J_i$, it contains: number of completed operations/total operations, slack time: $\left(d_i-t\right)-\mathrm{r}\mathrm{e}\mathrm{m}\mathrm{a}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{n}\mathrm{g} \mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l} \mathrm{p}\mathrm{r}\mathrm{o}\mathrm{c}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g} \mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}$, current job status, and set of available machines for the current process.
• Global State Vector ($s_t^{\mathit{global}}$): System time $t$, average machine utilization, average queue length, and proportion of overdue jobs.

This state space is encoded through a Graph Neural Network (GNN) for feature extraction, obtaining a low-dimensional representation $h_t=f_{\mathit{extract}}\left(s_t\right)$.

To exploit the inherent relational structure of the scheduling system, we model the production state as a heterogeneous graph ${\mathcal{G}}_t=\left(\mathcal{V},{\mathcal{E}}_t\right)$. The node set $\mathcal{V}={\mathcal{V}}^M\cup {\mathcal{V}}^J$ consists of machine nodes $v_i^M\in {\mathcal{V}}^M$ and job nodes $v_j^J\in {\mathcal{V}}^J$. Each machine node is associated with the feature vector from $s_t^{\mathrm{machine}}$, and each job node with the feature vector from $s_t^{\mathrm{job}}$. An edge $\left(v_i^M,v_j^J\right)\in {\mathcal{E}}_t$ exists if machine $i$ is capable of processing the current operation of job $j$.

We adopt a widely used two-layer GNN that performs message passing over ${\mathcal{G}}_t$. Let $h_v^{\left(l\right)}$ denote the feature vector of node $v$ at layer $l$ (with $h_v^{\left(0\right)}$ being its input feature).

The update rule for each node $v$ in layer $l+1$ is:

$$h_v^{\left(l+1\right)}=\sigma \left(W^{\left(l\right)}\cdot \mathrm{CONCAT}\left(h_v^{\left(l\right)},\hspace{0.25em}\mathrm{AGGREGATE}\left\{h_u^{\left(l\right)}|u\in \mathcal{N}\left(v\right)\right\}\right)\right)$$

(10)

where:

$\mathcal{N}\left(v\right)$ is the set of neighbors of node $v$ in ${\mathcal{G}}_t$;

$\mathrm{AGGREGATE}\left(\cdot \right)$ is a permutation-invariant function, we use mean pooling;

$W^{\left(l\right)}$ is a trainable weight matrix for layer $l$;

$\sigma \left(\cdot \right)$ denotes the ReLU activation function.

The first GNN layer $\left(l=0\right)$ transforms the raw node features and aggregates information from immediate neighbors. The second layer $\left(l=1\right)$ further refines these representations and outputs the final node embeddings. After the second layer, we read out the embedding of a designated global context node (which is connected to all machine and job nodes) as the overall state representation $h_t$. This graph-based encoding enables the DRL agent to capture the complex dependencies between jobs and machines, providing a more structured and generalizable perceptual foundation than a flat feature vector.

The proposed state representation provides the foundational information necessary to maintain system liveness. By explicitly including real-time machine status (e.g., idle, busy, failed) in $s_t^{\mathrm{machine}}$ and precise job progress in $s_t^{\mathrm{job}}$, the agent obtains a global view of resource contention and operational dependencies. This comprehensive visibility enables the agent to evaluate and select only those actions that maintain forward progress of all jobs. Therefore, this design of the state space effectively prevents the agent from making sequences of decisions that would lead to a circular wait condition, thereby ensuring that deadlock does not occur in the scheduled system.

5.1.2. Action Space Design (Action Space $\mathcal{A}$)

The action $a_t\in \mathcal{A}$ is defined as selecting an operation $O_{ij}$ from the set of schedulable operations and assigning it to an available machine $M_k$. This is a hierarchical decision process involving job selection and machine assignment. Job selection refers to the agent first choosing the operation with the highest priority from all unscheduled operations, followed by machine assignment where the optimal machine is selected for that operation from the available machine set ${\mathcal{M}}_{\mathcal{i}\mathcal{j}}$.

To reduce the dimensionality of the action space, we employ a parameterized policy network ${\pi }_{\theta }\left(a_t|s_t\right)$, whose output layer dimension is $\left|\mathcal{O}\right|$ (the total number of operations), with each output node corresponding to a priority score for an operation. A probability distribution is obtained through a Softmax layer, and then an operation is selected via a greedy policy. Machine assignment is based on an evaluation network $Q_{\varphi }\left(s_t,M_k\right)$ that selects the machine with the highest value.

The inner DRL agent employs a Q-learning algorithm, updating its action-value function according to:

$$Q\left(s_t,a_t\right)\leftarrow Q\left(s_t,a_t\right)+\alpha \left[r_{t+1}+\gamma \underset{a^{\prime }}{\max }Q\left(s_{t+1},a^{\prime }\right)-Q\left(s_t,a_t\right)\right]$$

(11)

where $Q\left(s_t,a_t\right)$ is the action-value function for state $s_t$ and action $a_t$, $\alpha$ is the learning rate, $\gamma$ is the discount factor, $r_{t+1}$ is the immediate reward, and $\underset{a^{\prime }}{\mathit{max}}Q\left(s_{t+1},a^{\prime }\right)$ is the maximum expected future reward.

5.1.3. Reward Function Design (Reward Function $r_{\mathrm{t}}$)

The reward function design follows the Reward Shaping principle to alleviate the sparse reward problem and guide the agent towards learning the objective. The reward signal consists of the following components:

$$r_t=r_t^{\mathrm{complete}}+r_t^{\mathrm{utilize}}+r_t^{\mathrm{due}}+r_t^{\mathrm{penalty}}$$

(12)

The completion reward ($r_t^{\mathit{complete}}$) is given when a job is completed, proportional to the job’s priority $p_i$:

$$r_t^{\mathit{complete}}=\alpha \cdot p_i$$

(13)

The utilization reward ($r_t^{\mathit{utilize}}$) is granted when a machine $M_j$ finishes an operation, encouraging higher machine utilization:

$$r_t^{\mathit{utilize}}=\beta \cdot {\displaystyle \frac{\mathit{processing} \mathit{time}}{\mathit{total} \mathit{time}}}$$

(14)

The tardiness reward ($r_t^{\mathit{due}}$) is given when a job $J_i$ is completed, providing a reward/penalty based on its earliness/tardiness:

$$r_t^{\mathrm{due}}=\gamma \cdot \max \left(0,d_i-t\right)-\eta \cdot \max \left(0,t-d_i\right)$$

(15)

The penalty ($r_t^{\mathit{penalty}}$) is imposed for constraint violations (such as assigning to an unavailable machine). This immediate negative feedback for invalid assignments is a critical mechanism to prevent the system from reaching a deadlock state, as it actively discourages the agent from selecting actions that could create circular waiting conditions among jobs.

$$r_t^{\mathrm{penalty}}=-P$$

(16)

Here, $\mathsf{\alpha }, \mathsf{\beta }, \mathsf{\gamma }, \mathsf{\eta }$ are weight coefficients, which constitute an important part of the hyperparameters $\mathsf{\lambda }$ that need to be tuned by the outer-layer Bayesian optimization.

5.1.4. DRL Agent Architecture and Training Process

The inner-loop DRL agent in this study is designed to learn an optimized policy that maps observed system states to scheduling actions through continuous interaction with the simulation environment, thereby achieving adaptive decision-making under complex and dynamic conditions. The agent’s design and training revolve around a value-based deep network, whose core lies in utilizing the encoded system-state representation to learn the long-term value of scheduling strategies via sequential decision-making.

The agent’s network architecture integrates state perception and decision-generation functions. the perception module relies on the Graph Neural Network encoder detailed in Section 5.1.1, which transforms the structured system state $s_t$ into a semantically rich feature embedding $h_t$. This embedding is then fed into subsequent fully connected network branches for decision-making. Specifically, one branch estimates the state-action value function $Q\left(s,a;\theta \right)$, providing a long-term return assessment for each feasible scheduling action; the other branch outputs the overall value $V\left(s;\varphi \right)$ of the current state, used to evaluate the quality of the state. During training, action selection follows an ε-greedy policy to balance exploration and exploitation.

The training process aims to optimize the network parameters so that the agent can accurately predict action values and improve its policy. Training is based on interaction experiences collected from the simulation environment and uses the Q-learning algorithm for iterative updates. Each interaction yields a state-transition tuple $\left(s_t,a_t,r_{t+1},s_{t+1}\right)$, which is stored in an experience-replay buffer. During parameter updates, a batch of historical data is randomly sampled, and a target network is used to compute the temporal-difference target. The parameters of the main Q-network are then updated by minimizing the mean-squared-error loss. The parameters of the target network are periodically copied from the main network to ensure the stability of the learning target. This process is consistent with the update formula given in Section 5.1.2, and the introduction of experience replay improves data efficiency and training stability.

The agent’s learning is guided by the multi-objective reward function defined in Section 5.1.3. The reward signal integrates multiple requirements such as makespan, machine utilization, and due-date satisfaction. The intrinsic weights of the reward components are automatically adjusted by the outer Bayesian optimization, thereby ensuring that the policy learned by the agent aligns with the global optimization objectives.

5.2. Mathematical Model of the BO-DRL Collaborative Mechanism

The outer-layer Bayesian Optimization treats the entire training process of the inner-layer DRL as a black-box function $F$:

$${\lambda }^{*}=\mathit{arg}\underset{\lambda \in \Lambda }{\mathit{min}}\mathbb{E}\left[F\left(\lambda \right)\right]$$

(17)

where $\mathsf{\lambda }=\left[\alpha ,\beta ,\gamma ,\eta ,\mathit{learning} \mathit{rate},ϵ,\dots \right]$ is the set of hyperparameters, and $F\left(\lambda \right)$ represents the average final makespan obtained by running the DRL agent configured with hyperparameters $\mathsf{\lambda }$ on multiple validation scheduling scenarios.

BO models $F\left(\lambda \right)$ by constructing a surrogate model using Gaussian Process (GP):

$$F\left(\mathsf{\lambda }\right)\sim \mathcal{G}\mathcal{P}\left(\mathsf{\mu }\left(\mathsf{\lambda }\right),k\left(\mathsf{\lambda },{\mathsf{\lambda }}^{\prime }\right)\right)$$

(18)

where $\mathsf{\mu }\left(\cdot \right)$ is the mean function and $k\left(\cdot ,\cdot \right)$ is the covariance function (kernel function). The next evaluation point is then determined through the Expected Improvement (EI) acquisition function:

$${\mathsf{\lambda }}_{next}=\arg \underset{\mathsf{\lambda }}{\max }\mathrm{EI}\left(\mathsf{\lambda }\right)=\arg \underset{\mathsf{\lambda }}{\max }E\left[\max \left(0,F_{min}-F\left(\mathsf{\lambda }\right)\right)\right]$$

(19)

where $F_{min}$ is the current optimal value.

5.3. Efficient Rescheduling in Dynamic Environments

The framework adapts to dynamic environments through the following mechanisms:

• Real-time State Updates: Any dynamic event (such as new order insertion or machine failure) triggers immediate updates to the state $s_t$, enabling the DRL agent to respond based on the latest state information.
• Rapid DRL Response: The trained DRL policy network ${\pi }_{{\theta }^{*}}$ achieves extremely fast forward propagation speeds (at millisecond level), enabling real-time online scheduling.
• Continuous BO Learning: The system can periodically (e.g., monthly) collect new scheduling data and rerun the BO cycle to optimize hyperparameters $\mathsf{\lambda }$, allowing the DRL agent to continuously adapt to changes in the production environment.

6. Experimental Validation

6.1. Experimental Design Rationale and Algorithm Configuration

This section outlines the rationale behind our experimental design and details the parameter settings for all compared algorithms. The design is structured to ensure a rigorous, comprehensive, and fair evaluation of the proposed BO-DRL framework within a context reflecting the complexities of opto-mechanical manufacturing.

(a)

Benchmark Problem Selection: The core experiments utilize a bespoke 20 jobs × 20 machines FJSP benchmark. The choice of this specific scale is pivotal: as quantified by Equation (8), the solution space size for this instance exceeds ${10}^{200}$, signifying that the problem’s complexity transitions from the exponential to the hyper-exponential regime. This establishes the 20 × 20 instance as a formidable benchmark that truly challenges the limits of scheduling algorithms. Furthermore, its enrichment with domain-specific constraints (Section 3) ensures that this complexity is not merely combinatorial but also reflects the intricate feasibility rules of high-precision opto-mechanical manufacturing, providing a valid and stringent testbed for advanced algorithms.

(b)

Choice of Baseline Algorithms: Genetic Algorithm (GA) and Particle Swarm Optimization (PSO) are adopted as primary baselines. They are widely recognized as representative and high-performing meta-heuristics for combinatorial scheduling, providing a credible and standard reference point for performance comparison.

(c)

Comprehensive Performance Metrics: A multi-dimensional evaluation system is employed, encompassing makespan, mean flow time, machine utilization, number of tardy jobs, and robustness indices. This holistic approach aligns with practical manufacturing objectives and prevents over-optimization toward a single metric.

(d)

Fairness in Resource Allocation: All algorithms were allocated identical computational resources, including the maximum number of iterations and hardware environment. The parameters for each algorithm were independently fine-tuned through systematic preliminary experiments to ensure that each operated in its most competitive configuration. This approach guarantees that any observed performance differences are attributable to the advantages of the algorithms’ core mechanisms, rather than imbalances in resource allocation or parameter tuning.

(e)

Scalability and Robustness Tests: Systematic tests from 5 × 5 to 100 × 100 scales assess scalability.

These settings provided a balanced and competitive baseline for each algorithm, enabling a meaningful comparison of their core capabilities.

To ensure a fair comparison among the algorithms, all algorithms adopted the same stopping criterion (maximum iteration count of 1000) and maintained consistent computational resources.

(a)

BO-DRL Algorithm: Discount factor 0.99, experience replay buffer size 10,000, batch size 32, employing an ε-greedy strategy.

(b)

Genetic Algorithm (GA): Population size 50, crossover probability 0.8, mutation probability 0.1, tournament selection (size 3), utilizing order crossover (OX) and swap mutation.

(c)

Particle Swarm Optimization (PSO): Population size 50, inertia weight 0.7, individual learning factor 1.5, social learning factor 1.5, maximum velocity 0.2.

6.2. Algorithm Performance Comparative Analysis

6.2.1. Solution Quality Comparison

To evaluate the comprehensive scheduling performance of the algorithms, a thorough comparison of the BO-DRL, GA, and PSO algorithms was conducted across four key dimensions: makespan, mean flow time, machine utilization, and due date satisfaction. The results are summarized in

Table 5. Comprehensive comparison of algorithm solution quality.

Algorithm	Makespan (h)	Mean Flow Time (h)	Machine Utilization ± Std (%)	Number of Delayed Jobs
BO-DRL	69.32	62.16	55.03 ± 11.39	1
GA	80.02	68.62	50.23 ± 10.30	1
PSO	93.06	77.16	43.80% ± 14.82%	3

. This comparative presentation allows for a direct assessment of each algorithm’s effectiveness: shorter makespan and mean flow time indicate higher production efficiency; higher machine utilization reflects better resource exploitation; and fewer tardy jobs demonstrate superior due-date adherence. The data in Table 5 thus provides the quantitative foundation for concluding that BO-DRL achieves a more balanced and superior overall performance across these competing objectives.

The data analysis indicates that the BO-DRL algorithm achieves superior performance across all key metrics. Specifically:

The BO-DRL algorithm obtained a makespan of only 69.32 h, representing improvements of 13.37% and 25.51% compared to GA (80.02 h) and PSO (93.06 h), respectively. This result directly validates the exceptional capability of the BO-DRL framework in compressing the production cycle and enhancing overall output efficiency.

BO-DRL also achieved the best mean flow time (62.16 h) among the three algorithms, indicating its ability not only to complete the last job quickly but also to accelerate the flow of all jobs holistically, effectively reducing work-in-process inventory.

Regarding machine utilization, BO-DRL reached 55.03%, significantly higher than GA’s 50.23% and PSO’s 43.80%. Crucially, while achieving higher utilization, its standard deviation (11.39%) was comparable to GA (10.30%) and substantially lower than PSO (14.82%). This indicates that through its intelligent decision-making mechanism, BO-DRL can more precisely balance the system load, avoiding both excessive idling of some machines and bottlenecks on others, thereby achieving more stable and efficient utilization of system resources.

In terms of the number of tardy jobs, both BO-DRL and GA recorded only 1, outperforming PSO’s 3. Combined with the shorter mean flow time and makespan, this indicates that while optimizing time-related metrics, the due-date awareness integrated into BO-DRL’s reward function enables it to intelligently assign higher priority to critical or time-sensitive jobs, thus better meeting due date requirements at a global level.

Gantt charts were obtained from the same experimental runs that produced the quantitative metrics in Table 5. A comparison of the charts reveals distinct structural differences corresponding directly to the measured performance. The schedule generated by BO-DRL (Figure 4a) exhibits minimal idle gaps between consecutive operations and a more continuous allocation of tasks across machines. This compact profile visually translates into the shorter makespan and higher utilization recorded in Table 5. In contrast, the schedules produced by GA (Figure 4b) and PSO (Figure 4c) show pronounced machine idle periods (visible as white segments) and longer waiting intervals between operations, which correspond quantitatively to the longer makespan, lower utilization, and greater load imbalance reported in the table. The visual and quantitative representations are therefore consistent and mutually reinforcing: the structural compactness observed in the BO-DRL Gantt chart underlies its superior performance across all measured metrics.

The comprehensive superiority of BO-DRL in solution quality fundamentally stems from the synergistic enhancement effect between DRL and BO. The inner-layer DRL agent, through end-to-end training, learns to make sequential decisions under complex constraints. Its state space design enables the perception of global information, thereby facilitating decisions superior to those derived from local heuristic rules. The outer-layer BO ensures the DRL agent operates under the optimal configuration through automated and efficient hyperparameter optimization, overcoming the suboptimality inherent in manual tuning. In contrast, traditional metaheuristic algorithms (GA, PSO), when confronted with highly constrained, nonlinear problems such as opto-mechanical automated manufacturing, struggle to explore the solution space effectively due to their inherent search mechanisms, easily becoming trapped in local optima, which leads to the observed performance gap.

6.2.2. Convergence Performance Analysis

Convergence performance is a key metric for evaluating the practical utility of an algorithm. Figure 5 illustrates the convergence curves of the three algorithms on a 20 × 20 scale instance, showing the variation in makespan with the number of iterations.

Figure 5a illustrates the convergence characteristics of the three algorithms on the 20 × 20 benchmark. The plotted curves show that BO-DRL achieves convergence at the 341st generation, reaching its performance plateau substantially earlier than GA (902 generations) and PSO (976 generations), corresponding to a 62.24% and 65.09% faster convergence speed, respectively. This indicates that the search direction of BO-DRL is more targeted and efficient. Its smooth, monotonically decreasing curve without significant oscillation signifies a highly stable optimization process based on policy gradients. In contrast, PSO, despite a rapid initial decrease, quickly becomes trapped in a local optimum, showing limited improvement in subsequent iterations. GA converges slowly, and its curve exhibits considerable fluctuation, reflecting the blindness of its crossover and mutation operations.

The computational efficiency advantage of BO-DRL is quantified in Figure 5b, which compares the total computation time required for a complete optimization run. Under identical computational resources, the total computation time required for BO-DRL to complete one full optimization was reduced by 77.45% and 89.28% compared to GA and PSO, respectively. This significant advantage stems from the intrinsic mechanism of the BO-DRL algorithm: once trained, its forward inference for decision-making occurs at the millisecond level. In contrast, GA and PSO require expensive fitness evaluations (i.e., complete scheduling simulations) for the entire population in each iteration. This makes BO-DRL exceptionally practical for dynamic environments requiring rapid response or frequent rescheduling.

The superior convergence performance of BO-DRL can be attributed to three main factors:

• Directed Policy Search: Unlike the random search of metaheuristic algorithms, DRL performs directed policy improvement via policy gradients, continuously adjusting decisions towards actions that yield higher cumulative reward, naturally leading to higher search efficiency.
• Attention Mechanism: As described in Section 5.1.1, the attention mechanism within the state encoding network enables the agent to focus on the most critical scheduling decisions at any moment (e.g., bottleneck machines, urgent jobs), avoiding redundant searches on non-critical decisions. This is a key reason for its ability to quickly escape local optima.
• BO Preheating Effect: The outer-layer Bayesian optimization provides the DRL agent with a near-optimal initial hyperparameter configuration. This gives the inner-layer DRL training a higher starting point, equivalent to a high-quality “algorithm preheating,” significantly reducing the time required for convergence.

In summary, BO-DRL outperforms not only in final solution quality but also demonstrates substantial advantages in convergence speed, stability, and computational efficiency. This establishes a solid foundation for its deployment in industrial APS systems with high real-time requirements.

6.2.3. Algorithm Robustness Verification

To validate the robustness of the algorithms, we introduced three types of dynamic disturbances into the original benchmark instance. The specific parameters and application dynamics for each perturbation type were designed to reflect realistic production scenarios and are detailed as follows:

(a)

Machine Failures: We randomly selected 5 out of the 20 machines (25% of the total fleet) to simulate unplanned breakdowns. This failure rate represents a moderate-to-high stress scenario for the system. Each failed machine became unavailable for a duration uniformly distributed between 2 and 8 h, after which it resumed operation. Failures were triggered at random time points after the 20th hour of the schedule to simulate mid-production disruptions, ensuring the initial schedule was already in execution.

(b)

Urgent Orders: We inserted 3 new high-priority jobs during the scheduling process. These jobs were released into the system at random times uniformly distributed between the 10th and 30th hours. To reflect their urgency, each was assigned a due date tightness factor of 0.3 (i.e., due date $=\mathrm{release} \mathrm{time}+0.3\times \mathrm{total} \mathrm{processing} \mathrm{time}$), which is significantly tighter than the average factor of 1.2 used for regular jobs in the benchmark. Their internal process plans and machine eligibility were generated with the same complexity distribution as the original benchmark jobs.

(c)

Processing Time Fluctuations: To simulate natural variability in operation execution, the actual processing time for every operation was subject to a random fluctuation. The realized time was set to $p_{ijk}^{\mathrm{actual}}=p_{ijk}\times \left(1+\delta \right)$, where $\delta$ was drawn from a uniform distribution over the interval $\left[-0.15,+0.15\right]$, representing a ±15% variation. This range captures typical variability observed in manual adjustment and precision assembly stages.

These three perturbations were applied concurrently in a single, integrated dynamic scenario to test the algorithm’s ability to handle compound uncertainties. The specific parameters (5 machines, 3 urgent jobs, $\pm 15\%$ variation) were chosen to represent a significant yet realistic stress level for the scheduling system, providing a stringent and comprehensive test for robustness.

The performance of the algorithms under dynamic disturbances is quantitatively compared in Figure 6, which illustrates the makespan increase rate, rescheduling success rate, and solution quality retention rate.

Under these dynamic conditions, BO-DRL demonstrates exceptional adaptability. Its makespan increased by only 8.7%, and it achieved a rescheduling success rate of 94.2%, significantly outperforming the comparative algorithms. This robustness stems from the closed-loop nature of the DRL-based controller, which continuously perceives the state and reacts. The pre-tuning via BO ensures that the controller’s adaptation policy is near-optimal from the outset.

6.3. Algorithm Scalability Analysis

To comprehensively evaluate the performance of the algorithms across problems of varying scales, we conducted systematic tests on six additional opto-mechanical automated manufacturing benchmark instances ranging from 5 × 5 to 100 × 100. In response to the reviewer’s request for detailed specifications, the complete machine categorization, corresponding quantities, and task sequencing information (Gantt charts) for all these scales are provided in the Supplementary Material. The detailed comparison of key performance metrics for BO-DRL, GA, and PSO across these instances is summarized in

Table 6. Algorithm scalability analysis.

Problem Scale		5 × 5	10 × 10	15 × 15	20 × 20	25 × 25	50 × 50	100 × 100
Makespan (hours)	BO-DRL	46.18	69.54	69.31	69.32	73.77	90.19	102.46
	GA	46.55	71.79	75.38	80.02	89.44	114.95	134.73
	PSO	48.82	78.83	83.70	93.06	95.02	107.40	123.23
Mean Flow Time (hours)	BO-DRL	43.52	61.59	62.15	62.16	67.60	77.40	83.32
	GA	42.18	63.05	67.17	68.62	76.92	93.48	102.83
	PSO	44.98	69.07	75.94	77.16	75.51	87.13	98.49
Machine Utilization (%)	BO-DRL	62.61% ± 3.96%	54.98% ± 12.27%	54.42% ± 14.85%	55.03 ± 11.39	51.81% ± 10.60%	43.20% ± 13.34%	38.29% ± 15.42%
	GA	68.30% ± 12.02%	54.69% ± 5.48%	47.94% ± 10.49%	50.23 ± 10.30	43.22% ± 9.51%	33.97% ± 11.93%	29.17% ± 11.52%
	PSO	61.29% ± 17.04%	54.66% ± 12.00%	45.88% ± 12.14%	43.80% ± 14.82%	39.51% ± 11.88%	36.44% ± 14.21%	31.97% ± 14.21%
Number of Tardy Jobs	BO-DRL	0	1	2	1	3	13	45
	GA	0	1	2	1	10	32	75
	PSO	0	2	3	3	8	20	67
Disturbance Resistance (Solution Quality Retention Rate, %)	BO-DRL	96.3	94.7	93.1	91.5	89.6	85.2	79.1
	GA	90.2	88.9	86.8	84.8	82.5	76.9	69.8
	PSO	87.6	85.3	83.2	81.2	78.7	72.4	64.3

. This table reveals consistent trends: BO-DRL achieves the shortest makespan and mean flow time at every scale, maintains competitively high and stable machine utilization, and generally results in fewer tardy jobs compared to both GA and PSO. Notably, its relative advantage in makespan reduction becomes more pronounced as the problem size increases, underscoring its scalability.

Convergence behavior across these different scales is visualized in Figure 7. The side-by-side comparison for each scale clearly demonstrates that BO-DRL not only converges faster but also reaches a lower makespan plateau than both meta-heuristics. In contrast, the convergence profiles of GA and PSO frequently exhibit more erratic improvement patterns and a clear tendency to stagnate at higher objective values, particularly as the problem size and complexity increase in larger instances.

The progression of BO-DRL’s performance advantage is quantified in Figure 8, which charts its relative makespan improvement against GA and PSO across all problem scales. Performance improvement trend analysis demonstrates that the BO-DRL algorithm exhibits significant advantages across problems of different scales. Specifically, the performance improvement of BO-DRL relative to GA increases monotonically from 0.80% at the 5 × 5 scale to 23.96% at the 100 × 100 scale, showing a continuously strengthening trend. This phenomenon indicates that as problem complexity increases, the advantages of BO-DRL’s deep perception and adaptive decision-making capabilities become increasingly prominent.

As the problem scale increases, BO-DRL’s performance advantage becomes more pronounced. In the 25 × 25 large-scale instance, BO-DRL achieved an 11.6% improvement over GA, demonstrating its excellent scalability and applicability to more complex real-world production environments. In comparison, the performance improvement of BO-DRL relative to PSO shows a different pattern: it continuously increases for small to medium scales (5 × 5 to 20 × 20), reaching a peak of 25.51% at the 20 × 20 scale, then stabilizes around 16–17% for larger-scale problems. This trend reflects the relative adaptability of the PSO algorithm within specific scale ranges, while BO-DRL maintains a clear advantage across all scales.

Key findings include:

• Scale Adaptability Differences: The improvement of BO-DRL over GA continuously strengthens with increasing scale, indicating its stronger adaptability in complex large-scale problems, while its improvement over PSO stabilizes after peaking at medium scales, reflecting the characteristic differences of different algorithms when dealing with problems of varying complexity.
• Convergence Performance Advantage: For small to medium-scale problems, BO-DRL’s convergence speed significantly outperforms comparative algorithms, achieving stable solutions on average 60% earlier. For large-scale problems, BO-DRL effectively focuses on key scheduling decisions through its attention mechanism, avoiding redundant searches in invalid solution spaces.
• Disturbance Resistance Capability: As the problem scale increases, the disturbance resistance performance of all algorithms decreases, but BO-DRL shows the least degradation. For the 100 × 100 ultra-large-scale problem, BO-DRL’s solution quality retention rate is significantly higher than GA (+9.3%) and PSO (+14.8%), demonstrating its exceptional robustness.

In summary, the BO-DRL algorithm demonstrates excellent scalable performance, with its advantages becoming more evident as the problem scale increases. This is primarily attributed to its bi-level optimization architecture: the inner-layer DRL achieves adaptive decision-making through deep perception, effectively handling the complexity of large-scale problems; the outer-layer BO ensures the algorithm operates in an optimal configuration across different scales through efficient hyperparameter optimization. This design makes BO-DRL an ideal choice for solving large-scale scheduling problems in complex manufacturing environments.

6.4. Discussion and Insights

The exceptional performance of BO-DRL in complex opto-mechanical automated manufacturing scheduling can be attributed to the following key factors:

• Adaptive Decision-Making Capability: Through deep reinforcement learning, BO-DRL can adaptively adjust scheduling strategies based on real-time states, rather than relying on fixed heuristic rules.
• Constraint Handling Capability: The attention mechanism enables the algorithm to effectively identify and satisfy the complex process constraints in opto-mechanical automated manufacturing.
• Hyperparameter Optimization: Bayesian optimization ensures that the DRL algorithm always operates under the optimal hyperparameter configuration, fully realizing its learning potential.
• Knowledge Accumulation and Transfer: Through curriculum learning, BO-DRL can transfer knowledge learned from simple problems to solve complex problems.

In conclusion, the experimental results demonstrate that the BO-DRL algorithm significantly outperforms traditional metaheuristic algorithms in complex opto-mechanical automated manufacturing scheduling problems, showing clear advantages in solution quality, convergence speed, and robustness. It provides an effective solution for intelligent scheduling in complex manufacturing environments.

7. Discussion and Limitations

While the proposed BO-DRL framework demonstrates significant advantages in adaptive scheduling for opto-mechanical manufacturing, it is important to acknowledge its current limitations and scope to provide a balanced view and guide future work.

(a)

Computational Overhead of Offline Training. The initial phase of training the DRL agent and optimizing hyperparameters via BO is computationally intensive. Although the trained agent operates with millisecond-level latency online, this upfront cost must be considered for deployment scenarios where rapid adaptation to a completely new production configuration is required. Future work could explore meta-learning or transfer learning techniques to reduce this cold-start cost.

(b)

Dependence on Simulation Fidelity. The agent’s policy is learned and tuned entirely within a simulated production environment. Its performance in practice is therefore contingent on the accuracy of the simulation model in capturing the dynamics and stochasticity of the real shop floor. Discrepancies between simulation and reality could lead to suboptimal decisions. Enhancing the simulation with digital twin technologies or incorporating online fine-tuning mechanisms are valuable directions.

(c)

Generalizability of Dynamic Disturbance Models. The robustness tests, while designed with realistic parameters, employ specific, pre-defined disturbance profiles (e.g., uniform distribution for downtime). The framework’s performance under unforeseen or more extreme disruption patterns warrants further investigation. Extending the state representation and reward function to handle a broader, less structured set of anomalies remains a challenge.

(d)

Interpretability of the Learned Policy. Like many deep RL-based controllers, the inner decision-making logic of the trained DRL agent is not easily interpretable to human planners. In high-stakes, high-precision manufacturing, a degree of explainability may be required for trust and adoption. Developing methods to explain or distill the agent’s policy into human-understandable rules is an important avenue for future research.

Addressing these limitations will be crucial for transitioning the proposed intelligent control framework from a validated prototype to a robust, trustworthy component of future autonomous manufacturing systems.

8. Conclusions and Future Work

In this study, we developed a multi-objective adaptive scheduling algorithm that integrates deep reinforcement learning with Bayesian optimization to address the complex scheduling requirements in opto-mechanical automated manufacturing. Experimental results demonstrate that the proposed algorithm outperforms traditional metaheuristic methods in solution quality, convergence speed, and robustness. It effectively handles dynamic disturbances and balances conflicts among multiple objectives, providing an efficient and reliable intelligent control solution for autonomous manufacturing systems in high-precision manufacturing environments.

Future research will pursue full autonomy and enhanced intelligence for scheduling systems. Our efforts will focus on three key directions: (1) introducing meta-learning mechanisms to enable rapid adaptation to new production scenarios, (2) developing data-driven dynamic perception systems to anticipate and respond to disruptions proactively, and (3) exploring preference-free multi-objective decision-making to autonomously balance conflicting goals. These advancements are crucial steps toward realizing self-evolving intelligent control systems for manufacturing.