Improvement of Delivery Reliability by an Intelligent Control Loop between Supply Network and Manufacturing

Featured Application

Semiconductor manufacturers in a globally distributed and complex supply network are challenged by deviations between forecasts and actual demand. These deviations lead to events that manufacturers need to react to ensure their delivery reliability. Therefore, based on such events, manufacturing schedules need to be adjusted automatically on the manufacturing control level to ensure a smooth order flow and high delivery reliability within their manufacturing system of a complex job shop. This contributes to resilient manufacturing sites within a supply network.

Abstract

Manufacturing companies operate in an environment characterized as increasingly volatile, uncertain, complex and ambiguous. At the same time, their customer orientation makes it increasingly important to ensure high delivery reliability. Manufacturing sites within a supply network must therefore be resilient against events from the supply network. This requires deeper integration between the supply network and manufacturing control. Therefore, this article presents a concept to connect supply network and manufacturing more closely by integrating events from the supply network into manufacturing control’s decisions. In addition to the requirements, the concept describes the structure of the system as a control loop, a reinforcement learning-based controlling element as the central decision-making component, and the integration into the existing production IT landscape of a company as well as with latest internet of things (IoT) devices and cyber-physical systems. The benefits of the concept were elaborated in expert workshops. In summary, this approach enables an effective and efficient response to events from the supply network through smarter manufacturing control, and thus more resilient manufacturing.

1. Introduction

The manufacturing industry faces enormous challenges and changes worldwide. Mass personalization leads to smaller lot sizes with a simultaneously increasing number of variants [1]. Furthermore, the market is changing from a seller’s market to a buyer’s market. To meet the growing importance of customer satisfaction, a high delivery reliability must be achieved [1,2]. How challenging this can be was particularly evident during the COVID-19 pandemic: quarantine and other restrictions were making deliveries unreliable. Entire supply networks were thus not able to fulfill their logistic goals [3]. Therefore, the market environment in which manufacturing companies operate is becoming increasingly volatile, uncertain, complex, and ambiguous [4]. Manufacturing companies interact with this environment by organizing in supply networks, establishing a deeper integration in their supply network as well as by using enabling technologies for advanced automation in their smart factories to face this complexity [5,6]. Hence, competition has shifted from single companies to supply networks [7,8].

However, in a volatile, uncertain, complex, and ambiguous environment such as this, planning accuracy decreases significantly with an increasing time horizon [9]. This leads to a rising number of events in the supply network companies need to react to [6]. Events can be classified by their origin, cause, life cycle stage of the supply network, occurrence, subject and decision-making horizon and they must therefore be counteracted individually [10]. Since events can result in substantial financial implications (positive and negative), it is essential for manufacturing companies to react effectively and efficiently to events in the supply network [8,11]. This increases the need for coordination across all units of the supply network to achieve a high robustness and adaptivity, and thus high resilience [12,13]. Manufacturing control, which has the task of implementing the production plan despite unforeseen events, is, therefore, of high and further increasing importance [2].

Consequently, this leads to the research question of how manufacturing control can be effectively linked to supply network events and incorporate them into its decisions on countermeasures to prevent losses in delivery reliability. Therefore, this article describes the development of a concept transferring control engineering principles to establish an event-driven interface between supply network and manufacturing. Furthermore, a possible application with its resulting obstacles and benefits is discussed. The work contributes to the knowledge base in supply chain management, Production Planning and Control (PPC) as well as organizational theory. First, by recognizing the value of supply network events and designing an appropriate systematic reaction in manufacturing. Second, by deriving requirements for such a deeper integration between supply network and manufacturing. Third, by developing an intelligent concept establishing a deeper and event-driven integration between the two. Fourth, by complementing existing strategies for planning and control using approaches of areas such as control engineering and Machine Learning (ML). Fifth, by describing and appropriate application example in semiconductor manufacturing as well as its expected obstacles and benefits.

The article is organized as follows. Section 2 presents relevant related work. Section 3 then describes the concept to link supply network events and manufacturing effectively and efficiently. A possible application is detailed in Section 4. Section 5 discusses obstacles and benefits of the approach. The conclusion and contributions of the article are outlined in Section 6, whereas future work is described in Section 7.

2. Related Work

Related work relevant for this article covers the current interaction of supply network and PPC (cf. Section 2.1) as well as approaches to optimize this interaction. These approaches for optimization are given by the areas of supply chain event management (cf. Section 2.2.1), closed-loop control (cf. Section 2.2.2) and ML (cf. Section 2.2.3).

2.1. Interaction of Supply Network and Production Planning and Control

A supply chain is a goal-oriented connection of sites and companies, formed to optimize the flows of goods and information from suppliers to end users [8,14]. The supply chain includes all functions involved in receiving and fulfilling a customer request whereas the goal of a supply chain is to generate value for the entire supply chain [6]. Supply chain implies that there is only one participant per stage. In fact, however, there could be several participants per stage: for example, several suppliers and customers of a manufacturer. Consequently, material and information can also flow between more than two parties [6,8]. Chains are therefore actually networks, which is why the term supply network is more appropriate and will be furtherly used in this article. Only when existing concepts from the literature are referred to, their term is used (e.g., supply chain management and supply chain event management).

Due to changing conditions in their environment (cf. Section 1), manufacturing companies are increasingly organizing themselves in supply networks. They expect this to lead to cost savings, increasing profitability, additional market opportunities, higher customer satisfaction and more agile processes [6,8,15]. Agility, in particular, requires that supply networks evolve from their original function of cost savings to combining the capabilities of partners [16]. Considering this, supply network integration aims at making the whole supply network more competitive through the value it adds overall [7]. Hence, competition has shifted from single companies to supply networks [7,8].

However, in integrated supply networks, often up to thousands of individual but coordinated decisions must be made per minute [17]. These have a strong influence on the profitability and success of the supply network [6]. In this context, the planning and decision-making tasks can be classified according to the planning horizon and the affected process of the supply network (cf. Figure 1). Long-term decisions usually have a time horizon of several years and affect the fundamental structure of the supply network. With a time horizon of one quarter to one year, mid-term decisions include activities to maximize efficiency considering the superordinate structure. On a weekly or daily basis, short-term decisions attempt to reduce uncertainty in the supply network and process incoming customer orders in the best possible way [6,17].

Across sites and companies, supply chain management aims to coordinate decisions on the flow of goods, information and money, in order to increase the competitiveness of the supply network and meet customer requirements efficiently [7,17]. For a specific site, it is the task of PPC to pre-plan the continuous production program for several planning periods. Furthermore, to derive the required materials and resources from that and to implement the production program despite unavoidable disruptions such as staff shortages, machine malfunctions, supply delays and rejects [2]. Manufacturing control is especially responsible for the latter in a rather complex environment, since the number of possible order sequences alone grows exponentially with the number of machines and orders in a manufacturing system [2,18,19].

Planning objects essentially give the link between PPC and supply network. Planning objects are, for example, production quantities or production dates that refer to future states of individual elements of the supply network [15]. Presently, common PPC methods mostly use simulation and analytic approaches based on the planning objects to find optimal schedules. However, several approaches research on complementing or replacing these methods by ML [20].

2.2. Approaches to Improve the Interaction of Supply Network and Production Planning and Control

2.2.1. Supply Chain Event Management

A software system is event-driven, when it reacts to received events and creates new events. Consequently, events may flow across all components of the system [21]. Therefore, an event is an occurrence within a particular system as well as a programming entity that represents such an occurrence in a software system, also referred to as “instance” [22]. Such an instance of an event can be understood as a message containing information about an activity in a system. Furthermore, it may be related to other events [21]. Typical events in manufacturing companies, among others, are placed or adjusted customer orders, changed states of machines or resource alarms [17,22,23]. Companies have therefore always been event-driven. Increasingly important, however, is the recognition of the value lying in events and their aggregation and correlation, inspired by the findings of software development [21]. In general, software-based reaction to real-world events is also referred to as complex event processing. In this context, complex event means that an event is the consequence of a multitude of previous events [21,22]. Since the impact of events on manufacturing companies especially in the case of disturbances can be severe, the applications of event-driven software approaches are diverse, also in supply chain management, logistics and production [24,25,26].

Supply Chain Event Management (SCEM) aims at identifying deviations between the actual and planned state in the form of events from the supply network and react to them. Events can be positive (e.g., placed customer order) or negative (e.g., resource alarm), and, in particular, consequences from negative events should be mitigated. Therefore, knowledge generated from events is intended to maintain the efficiency of the supply network and ensure customer satisfaction [17,27,28,29]. Events must therefore be identified in time to avoid consequences or, if this is not possible, to mitigate consequences of the events as much as possible through proactive measures [29]. SCEM is embedded in supply chain management: although SCEM is focused on the short-term, operational planning, management and control of measures to mitigate the consequences of events, the planning level of supply chain management tries to minimize the number of events [29,30].

In addition to being understood as a management concept, SCEM can also be understood as an event-driven software solution. Both perspectives go hand in hand and cannot be viewed separately from each other, as they influence each other significantly [27,28]. Thus, SCEM as an information system can complement supply chain management [28].

SCEM solutions consist of five core tasks [31] First, status monitoring of all relevant processes and comparison with predefined planned values. Second, active and timely notification when an event is identified. Third, the simulation of options for action as a possible reaction to an event. Fourth, the selection and initiation of the most appropriate action to eliminate the target-actual deviation. Fifth, the monitoring of performance after implementing the action. Although it was addressed in passing by Otto [27], SCEM is not focused on building learning systems. Instead, the selection of actions takes place based on defined rules for evaluating the simulation results.

The degree of automation of a SCEM solution can be very different and can range from simple monitoring systems to decision support systems to complex autonomous systems. The focus of the literature, however, is on decision support systems; autonomous systems have hardly been addressed in this context so far [32]. Applications for SCEM are described for various industries in the literature, e.g., in [33,34,35,36,37].

2.2.2. Closed-Loop Control

Closed-loop control, also referred to as feedback control, is a process in which the controlled variable x is continuously or sequentially measured, compared to the reference variable w and influenced by the manipulated variable y to adapt it to the reference variable [38,39]. One or several disturbance variables z can also be implemented. The objective of the controlling element is, within the controlled system, even with the occurrence of disturbances, to adapt the controlled variable to the reference variable. A characteristic of closed-loop control is the closed-loop action sequence with continuous self-influencing of the controlled variable in the action path of the control loop [39]. A distinction can be made between continuous feedback control and sampling control as well as between non-adaptive and adaptive control [39].

Adaptive control is necessary to couple systems directly and to adapt models to changing conditions. Therefore, the principle of closed-loop control has been successfully used for many years in manufacturing processes and has also received a great deal of attention in PPC [9,40,41]. Here, the principle of the control loop is transferred to elements of the PPC to achieve a continuous adaptation of manufacturing (cf. Figure 2).

An overview and classification of existing approaches for control loops in PPC is given by Niehues et al. [42]. This classification is supplemented by an industry standard concept for model-based process control [40].

Table 1. Classification of control loops in manufacturing (adapted from [43]).

Category According to [42]	Characteristic Representative
(1) Control of Inventory	Wiendahl [44]
(2) Control of Capacity	Begemann [45]
(3) Control of Load	Scholz-Reiter et al. [46]
(4) Control by Rescheduling	Brackel [47]
(5) Knowledge-based Control	Philipp et al. [48]
(6) Model-based Control	Sachs et al. [49]

shows a characteristic concept for each category.

According to [42], categories one to three are closely related by the manufacturing system’s load as their controlled variable. The load can be determined by controlling order releases and, therefore, the inventory (1), or by capacity adjustments (2), or by a more complex setup of multiple controlled and manipulated variables (3). Since categories four and five build on a terminated production plan, their control activities are only performed in case of deviations. The controller type can be distinguished between rescheduling the entire production plan (4) and knowledge-based and predefined procedures to react in the current manufacturing situation (5). The categories are complemented by an approach from discrete, model-based process control (6).

These concepts focus primarily on the internal optimization of manufacturing and not on the integration with the supply network. Company-external events from the supply network are consequently not considered. The concepts are also designed adaptively for known parameters only and not as learning systems. The degree of automation is mainly less and only a few concepts intensively consider multi-dimensional parameters as manipulated variables [43].

2.2.3. Machine Learning

A system can be described as learning if it is improving its performance on future tasks after making observations about its environment [50]. In such a system, ML aims at generating knowledge from data by developing a complex heuristic model from training data [51].

The automated development of models from data by ML is also the main difference to earlier approaches of artificial intelligence, which were based on manually constructed knowledge bases such as predefined rules in case-based reasoning [51,52,53]. Furthermore, ML also differs from mathematical optimization since it allows generalization to unknown data [54]. Therefore, the use of ML is particularly worthwhile when, due to the complexity of the problem, not all possible situations and all changes over time can be anticipated. This is also the case when, due to the complex relationships and interactions between parameters, it is unclear what the algorithmic solution to the problem must look like [50]. In manufacturing systems this is related to the combinatorial complexity and the dynamic complexity [55]. Experience has shown that ML is much more suitable than conventional methods of data analysis, especially for more than 15 dimensions in the data sets [56].

As described by Figure 3, ML methods can be categorized per type of feedback into unsupervised learning, supervised learning and Reinforcement Learning (RL) [50]. ML is especially suitable for tasks in manufacturing, since due to its complexity, manufacturing can behave unexpectedly when manual interventions are made; under- or over-steering is possible if parameters are incorrectly manually adjusted [2]. Each of the methods is suitable for different learning tasks in manufacturing, such as unsupervised learning for fault detection [57], supervised learning for process control [58] or load forecasting [59] and RL for order dispatching [60].

RL is especially suitable if labeled data is not available for a sequential learning problem and agents and their environments as well as reinforcements for actions can be modeled [50,61,62,63]. In RL, an agent learns a strategy of how to act to maximize a cumulative reward. This strategy is learned from a series of situations and taken actions, as well as their corresponding reinforcements, which may be either rewards (positive) or punishments (negative). In the case of punishments, reward maximization means that the agent strives to minimize the cumulative punishment. The agent is not told which actions to take, but instead must discover which actions yield the most cumulative reward by trying them. Therefore, RL is suitable, if a solution for a problem can be found by trial and error, meaning learned through receiving feedback from an environment. Additionally, it can handle delayed feedback [61,62]. As a prerequisite for the application of RL, it is necessary to describe the problem as a Markov Decision Process (MDP). In addition, a simulation model is most often necessary. This is used to enable the agent to perform the trial-and-error learning safely without interfering with a real environment [61,63].

The MDP describes states ${\mathit{S}}_{\mathit{t}}$ in a state space S, actions ${\mathit{A}}_{\mathit{t}}$ in an action space A and rewards ${\mathit{R}}_{\mathit{t}}$ for a learning agent interacting with its environment at each of a sequence of discrete time steps t (cf. Figure 4). At each time step t, the agent receives some representation of the environment’s state, $S_t\in S$, and on that basis selects an action, ${\mathit{A}}_{\mathit{t}}\in \mathit{A}$. One time step later $\mathit{t}+1$, and partially caused by its action, the agent receives a numerical reward, ${\mathit{R}}_{\mathit{t}+1}\in \mathit{R}$, and finds itself in a new state, ${\mathit{S}}_{\mathit{t}+1}$ [61].

A good overview of existing approaches using RL in PPC is given by Kuhnle et al. [64]. Most approaches focus primarily on order dispatching on machine level in rather simple environments and on maximizing resource related key performance indicators such as use. Since the approaches are aiming at different manufacturing principles and the exact RL design is usually not described, their comparability is limited.

3. Concept

After a definition of requirements (cf. Section 3.1), a concept is developed to effectively link supply network events and manufacturing by incorporating them into decisions on countermeasures to prevent losses in delivery reliability. This is based on the transfer of control engineering principles to establish an event-driven interface between supply network and manufacturing (cf. Section 3.2) as well as an RL-based controlling element to enable the control loop to be learning and adaptive (cf. Section 3.3). The concept can therefore be understood as a synthesis, further development and complementation of components and approaches described by related work (cf. Section 2). It differentitates from these approaches by the requirements defined below (cf. Section 3.1). Furthermore, an architecture of how to integrate the concept into a common manufacturing IT landscape is elaborated (cf. Section 3.4).

3.1. Requirements

Since there are many conceivable applications for control loops in manufacturing companies, the foremost requirement is the suitability for the application. It must allow the control of manufacturing with the aim of improving its key performance indicator delivery reliability and considering its influences. The necessity of this requirement results from the motivation to transfer control engineering principles to the interface between supply network and PPC. Strictly speaking, however, this is not about controlling orders at individual machines, but about optimizing the order flow through the entire manufacturing system in terms of delivery reliability (cf. Section 1).

For this purpose, due to their potential consequences, it is also necessary to consider events from the supply network, so-called external events, in the control loop and that appropriate measures to adjust manufacturing can be taken in response to the former (cf. Section 1 and Section 2.1).

The environment of manufacturing companies is rather complex, whereas the link between supply network and PPC is mostly based on static planning objects. To overcome this, the control loop must be designed in the sense of an adaptive control in such a way that its data-driven model is not only oriented to predefined situations, but also dynamically adapts to unknown situations and resulting events (cf. Section 2.1 and Section 2.2.2).

This adaptivity is determined by the desired learning capability of the controlling element. It enables the controlling element to develop new decisions from data sets and, contrary to mathematical optimization, generalize to unknown data (cf. Section 2.2.3).

Moreover, a high degree of automation is necessary. First, for the controlling element to learn new situations from data and, thereby, adapting all result variables to changed environmental conditions. Second, to decide for and implement measures in manufacturing without the downsides of manual interventions. The system limits can thus be continuously extended (cf. Section 2.2.3).

For this purpose, the controller must be able to determine multi-dimensional parameters and their dependencies as manipulated variables. The complexity of the problems in PPC alone determines this complex controller design with multi-dimensional manipulated variables (cf. Section 2.1).

Since the planning and control tasks in supply network and manufacturing are rather complex and interlinked, a seamless integration of the control loop into the existing manufacturing IT is required. Most preferably, this is achieved by a modular approach (cf. Section 2.1 and Section 2.2.2).

All requirements are summarized in

Table 2. Requirements for the concept to effectively link supply network events and manufacturing.

No.	Requirement
(1)	Suitability for the Application
(2)	External Events
(3)	Adaptivity
(4)	Learning Capability
(5)	Degree of Automation
(6)	Multi-dimensional Parameters
(7)	Seamless Integration

and will henceforth be referenced by their corresponding number.

3.2. Control Loop Design

The requirements are implemented with the developed concept of the Adaptive Control Loop for an Integrated Supply Network (ACSN). The ACSN is a discrete, event-driven, adaptive and learning control loop that integrates supply network with manufacturing, more precisely supply chain management with manufacturing control (cf. Figure 5, blue control loop). It aims at mitigating the consequences of events induced by the supply network in manufacturing (cf. requirements 1 and 2). This complements existing approaches for control loops in manufacturing (cf. Figure 5, green control loop) aiming at the mitigation of events induced by manufacturing (cf. Section 2.2.2). Therefore, a cascade control is formed, whereas state of the art forms the inner control loop and the ACSN the outer control loop.

Like every control loop, the ACSN consists of a controlling element, the ACSN controller, as well as a controlled system, formed by manufacturing control and manufacturing here. Additionally, operating and machine data acquisition serves as the measuring element.

The ACSN’s reference variable is the production plan, which was generated by production planning. Adapted by the ACSN controller in case of events induced by the supply network, the adapted schedule serves as manipulated variable to interact with sequence deviation of manufacturing control (cf. requirement 6). The ACSN controller’s action alternatives result from the advantage of job shop manufacturing with its flexible adaptation to different work pieces and their machining sequences. Orders waiting in front of individual machines can be swapped at any time. Even the actual disadvantage of job shop manufacturing of a long lead time is used to advantage here by creating a period of time for the ACSN controller to adapt the schedule [2].

Simultaneously, the adapted schedule is the reference variable for the inner control loop of the cascade control. Thus, the adapted schedule influences (via manufacturing control and manufacturing) the expected delivery dates of manufacturing orders serving as the controlled variable. Events from the supply network, such as deviations between forecasts and actual customer demand, are considered. These events can either be detected when they occur using established technologies such as statistical process control or be predicted in advance [43]. For effective processing, events could then be classified [10], before incorporating them into the ACSN controller’s decisions.

Related to the five core tasks of SCEM (cf. Section 2.2.1), the ACSN builds on previous systems for monitoring and event identification. They are expanded and supported by automated decisions for countermeasures in the form of adapted schedules to mitigate the consequences of events from the supply network as well as the subsequent performance monitoring (cf. requirement 5).

3.3. Reinforcement Learning-Based Controlling Element

The ACSN becomes adaptive and learning fusing RL in the controlling element (cf. requirements 3 and 4). Therefore, no strategy is given for deciding on actions, but the ACSN controller derives them from observing its environment, in this case, the effects of its decisions on expected delivery dates in manufacturing. Consequently, the system can learn without labeled data and to adjust even complex manipulated variables with many dependencies automatically. Furthermore, effects can be handled that are only visible after orders have been fully processed in manufacturing. The quality and performance of the ACSN controller’s decisions are increasing over time. To realize a RL-based ACSN controller, an initial design, a training phase as well as an application of the trained agent is necessary (cf. Figure 6).

As a foundation for RL, the MDP must be designed. To fully represent the environment’s status, the state space S needs to contain data on production plan, supply network events as well as manufacturing feedback data. Here, the expected delivery dates are the most important data points of the manufacturing feedback data. Actions in action space A describe the ACSN controller’s scope of action to influence the production plan and consequently create an adapted schedule as input for manufacturing control. The ACSN controller is rewarded for taking actions leading to increased delivery reliability and, thereby, increasing the supply networks resilience. The outcome of this setup is the agent forming the ACSN controller. This includes an initial set of hyper parameters to use within the agent’s algorithm. Furthermore, a simulation model of the manufacturing system for the agent to interact with safely in training is needed. Ideally, an existing simulation model of a digital twin can be used to represent the manufacturing system as realistically as possible. If no such model exists, it must be built. Guidelines for this can be found in the literature [65]. An interface between the simulation model and existing manufacturing IT systems is important, in order to work with data in the simulation model that is as close to reality as possible.

To train the agent, it continuously interacts with the simulation model of the manufacturing system to evaluate various actions and their respective rewards. In doing so, the agent must always tradeoff between exploitation (maximize reward) and exploration (discover environment) when selecting an action. Therefore, the agent must be given a strategy such as $ϵ-\mathrm{greedy}$ or $\mathrm{decaying}-ϵ-\mathrm{greedy}$ [61]. Consequently, training is an iterative process varying hyper parameters that must be repeated many thousands of times until the reward is maximized. However, in addition to statistical validation, the suitability of the agent for the problem at hand must be checked, in particular before transferring it to its application. Doing so, the simulation model is used to simulate the same scenario with and without intervention by the agent. If the agent performs better regarding delivery reliability, it can be transferred to the application.

Subsequently, the ACSN controller can be deployed to its application in a real manufacturing environment. Consequently, the agent no longer explores, but selects specific actions for exploitation. Instead of interacting with a simulation model, the agent uses manufacturing control as an interface to interact with the manufacturing system. Actions are performed by adapting control values in manufacturing control, states are constructed by data gathered from various manufacturing IT systems.

3.4. Embedding in Manufacturing IT Landscape

To enable the ACSN to support manufacturing control effectively it must be seamlessly integrated into the common manufacturing IT landscape of manufacturing companies (cf. requirement 7). The proposed modular approach (cf. Figure 7) was inspired by a more general event processing architecture described in the literature [66]. It has been adapted for the processing of events from the supply network as well as extended to provide a suitable basis for a RL-based controller. The building blocks of the architecture are oriented to the automation pyramid with manufacturing level, manufacturing control level and enterprise control level. A data layer and a digital twin supplement these. Regarding stakeholders, both external companies and internal users are considered. Consequently, interfaces could be either human machine interfaces or application programming interfaces.

The manufacturing level consists of the equipment and work centers, their sensors, actuators and IT interfaces. Since newer approaches to manufacturing control see dispatching as a decentralized task assigned to the work centers, this is taken into account accordingly in the architecture, in contrast to the classic automation pyramid.

Although the manufacturing level mainly consists of physical equipment, the manufacturing control level and enterprise control level are IT driven and, therefore, responsible for data processing. The manufacturing control level consists of operating data acquisition and machine data acquisition to establish a data-based link to the manufacturing level. This link needs to be modular, flexible, and well scalable, which could be achieved by techniques such as a manufacturing service bus. Furthermore, an integration layer such as a manufacturing service bus allows for an integration with Internet of Things (IoT) devices and Cyber-physical Systems (CPS) within a smart factory [67]. Additionally, the Manufacturing Execution System (MES) is included as the central system of manufacturing control. The ACSN controller supplements the manufacturing control level and integrates the ACSN (cf. Section 3.2) with its RL-based controlling element (cf. Section 3.3) into the architecture.

On the enterprise control level, the systems to control the company (enterprise resource planning) and supply network (supply chain management) are found. Consequently, this also represents the main interface to external companies.

The ACSN’s decision on how to respond to events is made at the manufacturing control level. However, data from the enterprise control level (events) and the manufacturing level (manufacturing situation) are explicitly taken into account and included in the decision. The decision-making process is further supported by the data layer and the digital twin. Going beyond the architecture of classical event processing systems, a learning system in the data layer—besides events—also needs the history of production plans, manufacturing states and its model parameters. These data are fed by the systems of the three levels mentioned above. The digital twin addresses the need for an RL agent for a safe environment to perform learning while at the same time use data and system behavior as close as possible to the real environment. Therefore, semantics and algorithms are used to preprocess data of the data layer for simulation models representing physical and functional characteristics of the manufacturing system.

4. Application

In this section, a possible application of the ACSN is presented. To start, the semiconductor industry is described (cf. Section 4.1) followed by an illustration of an exemplary process that responds to events from the supply network (cf. Section 4.2). Then it is shown how the ACSN can practically support and optimize this process (cf. Section 4.3).

4.1. Semiconductor Industry

Semiconductor manufacturers are organized in globally distributed and complex supply networks with complex manufacturing sites [68,69]. In the supply networks various partners searching for their own local optima are involved. Flows of material, information and finance across different continents must be successfully managed. Additionally, cultures, customs and time zones must be organized [69,70]. The complex manufacturing process in this industry’s so-called wafer fabs, regarding their manufacturing principle also referred to as complex job shop, often containing more than 100 machines and up to 700 process steps results in a lead time of up to three months. Furthermore, it is characterized by criteria such as prescribed due dates of the orders, parallel machines, different process types, sequence-dependent setup times, frequent disturbances and re-entrant order flows [68,70]. Re-entrant order flows result in wafers that are at different stages of their manufacturing cycle having to compete for the same resources [70]. Therefore, even if complex job shops are a subtype of job shops, they differ significantly [18].

In supply networks such as this, planning accuracy decreases significantly with an increasing time horizon [9]. A study with 52 multinational semiconductor manufacturers with an annual turnover exceeding USD 400 million has shown that insufficient forecasts and their implications are the top supply network concern causing for example unnecessary high inventory levels and unreliable deliveries [71]. Even if modern approaches allow for improved forecasts, they are still forecasts with biases and errors [72]. Resulting deviations between forecasts and actual demand lead to many events that require appropriate responses [10].

4.2. Existing Process for Responding to Supply Network Events

The event response process must be clearly distinguished from the daily business process for demand planning between supply network and manufacturing. Although the daily business process controls the flow from company-wide demand planning to the release of lots in manufacturing and is therefore automated in many stages, the response to events from the supply network describes an escalation management and, therefore, includes various process steps that are carried out by humans. Both processes were modeled and analyzed in workshops with experts from various departments of different semiconductor manufacturers. Since the ACSN aims at optimizing the response to events from the supply network only an exemplary existing event response process is shown in Figure 8 and described below.

If an event occurs in the supply network, it must first be detected and communicated. Although this communication process could be automated, it is often the supply network planner who is the recipient of this event-related communication. The supply network planner then investigates what effects the event has on quantities or delivery dates and then coordinates with the responsible production planner. Communication between these two can take place in personal conversations, by telephone, email or supported by manufacturing IT (cf. Section 3.4). The production planner then examines the possibility of changing quantities or delivery dates. To do so, he uses manufacturing data consisting of the current work in progress, the remaining lead times, the available capacity and certain master data to reach a decision on countermeasures to mitigate the consequences of the event occurred in the supply network. The experience knowledge of the production planner also influences this decision. Subsequently, the supply network planner receives feedback as to whether the change in quantity or delivery date can be implemented in manufacturing. Again, not necessarily in an automated manner.

If the change in quantity or delivery date can be implemented, the production planner communicates certain countermeasures to adapt the schedule to the line manager in manufacturing. Also, in this case, this communication can take place in personal conversations, by telephone, email or supported by manufacturing IT. The line manager then incorporates the adapted schedule into the manufacturing control system, typically a MES (cf. Section 3.4). It should be noted that communication between the production planner and the line manager here is usually only unidirectional. The line manager does not inform the production planner, in particular not with accuracy to single lots, whether the decision on an adapted schedule was appropriate regarding manufacturing. Thus, there is no systematic exchange of information to improve future decisions.

Even though the semiconductor industry is without any doubt a high-tech industry, our workshop findings show that the escalation management process for events from the supply network in particular is often characterized by media discontinuities and manual decisions at many stages. These media discontinuities can lead to time-delayed communication and thus a shorter period of action for countermeasures. In addition, escalation management keeps people involved from focusing on daily business. Furthermore, due to the complex nature of the manufacturing process manual decisions can lead to unexpected behavior in manufacturing.

4.3. Application of the ACSN to Optimize this Process

Great potential is seen by integrating data from the supply network into decisions on the manufacturing control level [73]. This aims at improving delivery reliability, representing the second most supply network concern in the study on the semiconductor industry by Kremers [71] (cf. Section 4.1). Therefore, the semiconductor industry, more precisely its response to supply network events in manufacturing, is a well-suited application for the ACSN. Following this integration approach, the ACSN is designed to directly link events from supply network to manufacturing control in a complex job shop environment. The resulting optimization of the process for responding to events (cf. Section 4.2) is shown in Figure 9 and described below.

Assuming that events from the supply network are detected and communicated automatically, the ACSN can complement and partially substitute the described process from the supply network planner to the MES (cf. Section 4.2). For this purpose, the supply network planner continues to be informed about events that occur, but further communication and the decision on countermeasures are taken over by the ACSN.

As a first step, events occurring in the supply network are communicated to the ACSN. On the IT side, this is done via an interface between supply chain (event) management and the ACSN controller. Subsequently, the ACSN controller takes over the coordination and decision-making tasks that were previously distributed between the supply network planner and the production planner by automating the information flows as well as the decision-making process. However, the production planner still has the ability to intervene with the ACSN’s decisions. In addition to the interface to supply chain management already described, the automated decision-making process requires further interfaces within manufacturing IT landscape (cf. Section 3.4), which must be implemented to enable the ACSN to gain all relevant data enriching the supply network event with relevant context information.

The decision-making on countermeasures to adapt the schedule is automated by the RL agent of the ACSN controller. In addition to data on the event from the supply network and on the current situation in manufacturing, which the production planner would also take into account, the agent is also aware of the effects of countermeasures in manufacturing from its training phase. This assumes that the agent’s training necessary before deployment to its application has already been completed (cf. Section 3.3). Deploying the agent, the state and action variables of the agent must be mapped to the interfaces to the manufacturing IT landscape as described above. Consequently, a data-driven decision can be made at any time, which is not biased by possible subjective judgments of the production planner. The decision aims to define an adapted schedule for manufacturing, which can be implemented without negatively affecting the overall delivery reliability.

Once a decision on countermeasures to adapt the schedule has been made, these measures could be implemented automatically using the MES. Therefore, the interface to the MES in particular must also enable controlling access to implement the adapted schedule for mitigating events’ consequences in manufacturing. The coordination process with the line manager in manufacturing is no longer necessary, but the line manager continues to be informed about the changes made and still can intervene. In addition, the ACSN controller receives information about the effects of the implemented measures in manufacturing at any time via the interface to the MES. Future decisions can be further optimized using this feedback in continuous training parallel to operation.

The ACSN’s features can additionally be illustrated using the event of a requested change to a delivery date as an example – an event that occurs at least once a shift with 46% of the participants in the study by Pfund et al. [74]. The delivery date can either be postponed, which does not necessarily require countermeasures but allows manufacturing capacity to speed up other orders, or preponed (cf. Figure 10). In particular, preponed delivery dates can be challenging to ensure overall delivery reliability despite their realization to satisfy customer requirements. With the acsn, the production planner and the supply network planner do no longer have to time-consumingly negotiate possible countermeasures and their exact configuration as well as manually adjust the corresponding control values via the line manager. Instead, knowing their dependencies and effects countermeasures and even complex combinations of these can be designed automatically by the ACSN controller. Subsequently, changed control values even for many orders are set automatically via the interface to the MES. Therefore, the ACSN is in summary a suitable complement to forecasts, mitigating the consequences of their deviations to actual demand represented by supply network events effectively and efficiently.

5. Discussion

When applying the approach of the ACSN to an application as the one described in Section 4, some obstacles must be overcome before achieving its benefits.

First, the problem formulation as MDP as well as the simulation model necessary for the agent’s pre-deployment training phase require several assumptions and simplifications compared to reality. Furthermore, the design of the agent’s reward function may be biased by human experience. The deployment of the ACSN to real applications is therefore associated with the challenge of ensuring the agent’s performance despite changed conditions. As a preventive measure, the use of a digital twin (cf. Section 3.4) is suitable in the training phase, which simulates the system behavior as close as possible to the real environment. As a protective measure, intervals can be set for the control values of the ACSN controller, which may not be exceeded. In addition, after deployment, the ACSN can be used in a transition period as a control system instead of a closed loop, with decisions reviewed by the production planner prior to implementation.

Second, the concept of the ACSN is highly dependent on data from supply network and manufacturing including data availability and data quality. Consequently, even if the architecture allows for a modular integration into the existing manufacturing IT landscape, several interfaces to existing IT systems need to be designed and implemented. This is especially true for the typically complex manufacturing IT landscape of semiconductor manufacturers [68,75]. It is, therefore, highly recommended to use platforms and standardized interfaces (e.g., [67]) instead of developing individual interfaces for each IT system risking vendor lock-in effects.

Third, the link between supply network and PPC is presently essentially established by planning objects only. To make the ACSN’s approach work to its full extent, besides the need for effective event identification and communication within the supply network (e.g., [76]) data exchange across companies and manufacturing sites has to be enabled. Again, platforms and standardized interfaces are a promising technical solution. Events then continue to occur, but they are detected and processed at the earliest possible time, allowing for a longer period for countermeasures to take effect.

Fourth, data exchange is not only a technical obstacle. To enable data exchange across companies and manufacturing sites, the so-called silo thinking in supply networks must be broken down. A relationship of trust must therefore be established between the partners in a supply network, allowing extensive live data to be exchanged between them. How challenging this could be even within organizations as well as approaches to do so are discussed in the literature [77].

Fifth, there must also be a relationship of trust between employees and systems for automated decision-making. Among others, the reason for dispatching rules being so popular in semiconductor manufacturing is that their decisions are easy to understand [78]. To build up trust such as this and thus increase acceptance among employees, it is useful to give the decisive employees, especially in manufacturing control, the opportunity to monitor and override the ACSN’s automated decisions at any time. The emerging field of explainable artificial intelligence may offer additional strategies for transparent decisions and thus acceptance in the future [79].

However, after overcoming these obstacles, multiple benefits could be expected by the ACSN as they have been elaborated in workshops with experts from the areas of supply network planning, PPC and manufacturing of a semiconductor manufacturer. These benefits can be categorized by their relation to the event responding process itself, to decision-making, to manufacturing IT landscape and to the supply network.

It is expected that by automating the process of responding to supply network events in manufacturing control, the ACSN will enable a more efficient and a more effective overall process. On the one hand, this is due to the prevention of media discontinuities and, therefore, less sources of errors regarding data transfer. On the other hand, less media discontinuities also massively accelerate communication. In addition, the process also becomes more efficient because fewer people are involved, thus eliminating time-consuming coordination processes. Consequently, people in supply network and manufacturing departments can focus on their daily business processes rather than time-consuming and unpredictable escalation management. Additionally, the ACSN allows for an optimization of PPC–which has not been in scope so far–by deriving measures for mitigating the consequences of supply network events on manufacturing control level.

Furthermore, by knowing effects and dependencies of countermeasures from the training phase RL will make the decision on countermeasures in manufacturing control more robust by avoiding errors that occur when manual interventions are made–under- and over-steering could be avoided. Even complex combinations of countermeasures and their associated control values can be designed automatically. Due to their combinatorial complexity and dynamic complexity mathematical optimization reaches its limits in complex job shops very fast. Since the use of RL in the controlling element allows for generalization to unknown situations it is, therefore, ideally suited here. Additionally, since a simulation-based training phase is used, RL in particular has the decisive advantage that no elaborately labeled data need to be available. Furthermore, with its solely data-driven decisions the ACSN avoids biases usually brought in by human decision-makers and reduces the selection of wrongfully chosen countermeasures due to its direct feedback-loop with manufacturing. Moreover, the ACSN increases its performance over time through continuous training in parallel with its operation. In addition, to increase acceptance, it is possible to override the ACSN’s decisions at any time–production planner and line manager always have the ability to intervene. Furthermore, the scalability of the ACSN must be highlighted. Besides scalability regarding data variety–as described by the generalization to unknown situations–scalability is also necessary regarding data volume, since the processing of large amounts of data has been standard for many years in the semiconductor industry. However, large amounts of data must be processed especially during the pre-deployment training phase. In this phase, however, processing can be scaled to multiple resources through distributed learning. After deployment, decisions can be executed with comparatively little data and very quickly.

Furthermore, the system architecture of the ACSN enables a seamless integration by using standardized interfaces of an integration layer to connect to common manufacturing IT landscapes as well as to IoT and CPS. This allows for the implementation of even complex combinations of countermeasures since their associated control values can be orchestrated with the MES automatically. Just as importantly, the ACSN also improves the database for other systems of the manufacturing IT landscape.

Most importantly, the manufacturing site as well as the whole supply network benefit from the ACSN. With the ACSN’s decision-making process it is possible to improve delivery reliability despite the occurrence of numerous supply network events. Consequently, customer satisfaction increases with the improved delivery reliability, which is of great importance regarding the challenging environment of manufacturing companies. In addition, significant cost reductions can be achieved, since the deeper integration of supply network and manufacturing alongside with the effective mitigation of supply network events allows safety stocks to be reduced. Furthermore, by a more intelligent manufacturing control, the approach of the ACSN allows for an effective and efficient response to events from the supply network, thus a more resilient manufacturing.

6. Conclusions and Contributions

The market environment in which manufacturing companies operate is becoming increasingly volatile, uncertain, complex, and ambiguous. This increases the need for coordination across all units of their supply network to achieve high resilience. Therefore, this article describes our concept developed to effectively and efficiently link supply network and manufacturing control by incorporating supply network events into manufacturing control’s decisions on adapted schedules. This aims at improving delivery reliability and resilience. Control engineering principles are transferred to establish an event-driven interface between supply network and manufacturing. The ACSN as the resulting control loop becomes adaptive and learning fusing RL in the controlling element. Additionally, by its modular approach the ACSN can be seamlessly integrated into a common manufacturing IT landscape.

Our work contributes to the knowledge base in supply chain management, PPC and organizational theory. First, by recognizing the value of supply network events and designing an appropriate systematic reaction in manufacturing. Manufacturing control then is no longer limited to internal disturbances considered to be events, but instead systematically complemented to include external events from the supply network. Second, by deriving requirements for such a deeper integration between supply network and manufacturing. Besides the suitability for this application considering supply network events in manufacturing control for complex job shops, these requirements are adaptivity and learning capability of the controlling element, a certain degree of automation, the capability to determine multi-dimensional parameters as control values and a seamless integration with existing manufacturing IT landscape. Third, by developing an intelligent concept establishing a deeper and event-driven integration between supply network and manufacturing. This is achieved by the concept of the ACSN as a discrete, event-driven, adaptive and learning control loop. Fourth, by complementing existing strategies for planning and control using approaches of areas such as control engineering and ML. On the one hand, the ACSN is designed as cascade control to complement existing internal control loops in manufacturing. On the other hand, its RL-based controlling element allows for decisions on complex combinations of countermeasures and their associated control values. Fifth, by describing and appropriate application example in semiconductor manufacturing as well as its expected obstacles and benefits. Obstacles include transferring the decision model from a simplified training environment to a real manufacturing environment, the concept’s strong dependence on data, sharing data between partners in a supply network–technically and organizationally–and building employee trust in automated decision-making processes. However, after overcoming these obstacles, multiple benefits could be achieved by the ACSN. It accelerates the process for responding to supply network events by automating communication, involving less people and reducing media discontinuities. Furthermore, the data-driven decision-making not only allows for robust decisions without human biases, but also enables complex combinations of countermeasures to mitigate the consequences of supply network events. Most importantly, the whole supply network benefits from the ACSN by an ensured delivery reliability. Manufacturing sites are getting more resilient despite large numbers of supply network events.

7. Future Work

To extend knowledge in the research field and to further apply the ACSN, the authors are currently focusing on the following aspects. First, a simulation model based on the widely used dataset MIMAC set one [80] is implemented containing an interface for the training of RL-based agents on the manufacturing control level. Second, during the training phase, suitable algorithms and hyper parameters for the ACSN controller need to be evaluated and selected with regards to their suitability for the hedging of delivery reliability despite supply network events. Afterwards, the ACSN can be deployed to a real application allowing for a quantification of the benefits discussed in Section 5. Prerequisites and limits for the ACSN can then be derived and, subsequently, the ACSN eventually can be applied to further use cases.