Real-Time Maintenance Policy Optimization in Manufacturing Systems: An Energy Efficiency and Emission-Based Approach

Abstract

Within the frame of the Fourth Industrial Revolution, the application of Internet of Things technologies makes it possible to convert conventional manufacturing systems into cyber-physical systems, where the used new technologies enable the improvement of maintenance and operation processes. The aim of this work is to develop and validate a new real-time maintenance policy model and optimization algorithm based on digital twin simulation. The maintenance policy model is based on the conventional failure and operation data from ERP (enterprise resource planning) and the real-time and forecasted failure and operation data from digital twin simulation. The described maintenance policy model and its optimization algorithm represent an innovative way to manage predictive, preventive, corrective, and opportunistic maintenance strategies. The novelty of the presented method is that the real-time data generated by the digital twin solution allow the definition of a more accurate maintenance strategy. The optimization algorithm is characterized by a standard evolutionary algorithm. The impact of maintenance policy optimization on the energy efficiency and emission was analyzed in the case of both conventional and cyber-physical manufacturing systems with and without digital-twin-based simulation. The results showed that the energy consumption and the greenhouse gas emission in the real-time maintenance policy optimization scenario decreased by 21%, depending on the electricity generation source.

1. Introduction

The fulfilment of permanently increasing and dynamically changing customers’ demands requires new production concepts. The Fourth Industrial Revolution and its technologies open new directions to develop new production concepts [1,2]. The application of Industry 4.0 technologies led to the transformation of conventional production systems into cyber-physical production systems (CPPSs), which means the birth of the smart manufacturing systems (SMSs). In these complex manufacturing systems, the performance of operation and maintenance (O&M) plays an important role, because it is an important factor in quality control from the processes and products point of view and it can influence and determine the long-term success of value chains and companies [3,4]. The malfunctions and errors in manufacturing systems can influence the energy efficiency and lead to increased operation costs. Therefore, it is important to include energy aspects into the optimization of maintenance policies. As the literature review shows, there is a wide range of articles focusing on maintenance-related research topics, but only a few of them include sustainability, environmental, and energy-related aspects. Within the frame of this article, the authors propose a new maintenance policy optimization model, which makes it possible to supply the real-time maintenance policy optimization of cyber-physical manufacturing systems, where Internet of Thing (IoT) technologies are available. The digital-twin-enabled real-time maintenance policy optimization is a new research direction that focuses on the potential in digital twin technology, and it can be applied in the field of preventive, predictive, corrective, and opportunistic maintenance strategies. The proposed digital-twin-based model uses an integrated mathematical model, where the Markov decision process is based on value determination equations (VDEs) [5], and the NP-hard assignment problem is solved with an evolutionary heuristic algorithm.

As the below detailed literature review shows, a wide range of the literature focuses on the optimization of maintenance operations and strategies in the case of conventional manufacturing environments, and only a few of them focus on the potentials in the application of Industry 4.0 technologies. The three important factors of maintenance policy optimization are the followings: choosing maintenance standards, defining a suitable maintenance strategy, and optimizing the allocation of resources for maintenance operations. In the case of cyber-physical manufacturing systems, the standards are the same, as in the case of conventional manufacturing systems, but Industry 4.0 technologies make it possible to gather real-time information from the manufacturing and related logistics system and use them for a more sophisticated maintenance policy optimization, where real-time status information and failure data can also be taken into consideration. To try to fill this research gap, this article focuses on the development of a maintenance policy optimization methodology to analyze the potential of the transformation of conventional manufacturing systems into cyber-physical systems. The innovative aspect of this research is that different types of digital twin solutions can be used for this real-time data acquisition, including digital twin aggregate for processes, digital twin prototype for products and machines, and the digital twin environment, which integrates them into a cyber-physical system.

This paper is organized as follows. Section 2 presents a short literature review and shows the research gaps. Section 3 describes the model framework of real-time digital-twin-enabled maintenance policy optimization. In this section, both the functional and the mathematical models are described. The models focus on the real-time optimization of maintenance policies from the energy and emission point of view, while not only ERP and MES data, but also real-time failure data and status information are taken into consideration. Section 4 shows the computational results. Conclusions, managerial impacts, and future research scopes are discussed in Section 5.

2. Literature Background

Within the frame of this short literature review, the following questions are answered regarding digital-twin-based maintenance policy optimization: What is the current state of knowledge on digital-twin-based maintenance related research? What are the most important influencing factors of maintenance policy optimization? What are the main research gaps and what are the limitations of existing research results? The main results of this survey are shown in

Table 1. Contributions related to digital-twin-enabled maintenance policy optimization.

Key Feature of the Article	Energy	Real-Time	Maintenance	Optimization
Risk evaluation under fuzzy environment [4]			+	+
Distributed anomaly detection for maintenance [5]			+
Energy-consumption-based maintenance policy [6]
Lean-kaizen concept in maintenance [7]			+	+
Literature review [8,9,10,11]		+	+	+
IIoT-enabled production system [12]		+	+	+
DT in continuous lifecycle integration [13]	+	+	+
IoT sensors for consumption monitoring [14]	+	+	+
Sustainability audit for railway stations [15]	+	+	+	+
Data-driven ship digital twin [16]	+	+		+
Machine learning for railway asset management [17]			+	+
DT-based anomaly detection process flow [18]		+
Risks and benefits for a DT in maintenance [19]		+	+	+
Digital twin in energy conversion systems [20]	+	+	+	+
DT for buildings [21]		+	+
LCE for turbomachinery with DT [22]		+	+	+
Visualized inspection system for O&M [23]		+	+
Time series behavior modeling with digital twin [24]	+	+	+
DT as geodata hub for cities [25]	+	+
Data analysis in DT for production systems [26]		+
Digital twin in lifecycle management [27]	+	+	+
DT-based monitoring framework for OWT [28]		+	+	+
Human capital requirement for DT [29]		+
Application of DT in asset lifecycle [30]	+	+	+	+
DT for fatigue life prediction [31]		+
Industry 5.0 [32]	+		+
Human cyber-physical systems for O&M [33]	+	+	+
Low-cost DT for monitoring [34]		+	+
Cloud-based DT for smart manufacturing [35]		+		+
Secure embedded intelligence in O&M [36]	+	+	+
Real-time policy optimization	+	+	+	+

, including the highlights of content analysis of the related literature and the summary of the main contributions related to digital-twin-enabled maintenance policy optimization.

The main approaches regarding design methods are the following: application of fuzzy logic [1], lean approach including kaizen, poka-yoke [2], Markov decision process [5], Howard’s policy iteration [5,6], machine learning [17], simulation [17], digital-twin-based simulation [27], knowledge-based optimization [30].

The digital-twin-based simulation can support the decision-making process during both the design and operation of production and service processes. As a case study shows, a logistical simulation tool can support the design and operation phase of fusion power plants, where the duration of maintenance is predictable from a pre-operational simulation tool, which may support maintenance control based on physical data in real-time [19]. Digital twins can predict the future performance, behavior, and maintenance needs of a complex system. The digital-twin-enabled solutions can support fault diagnostics and prognostics in the industrial domain for a low-cost maintenance and higher utilization of technological, human, and logistic resources [20].

The application of digital twin solutions is especially important in the case of capital-intensive equipment, including building lifecycle management, but technical shortcomings of the state-of-the-art IoT technologies are also to be taken into consideration. A case study summarizes the results of a 25,000-sensor-based digital twin solution regarding reading, collection, computation, and utilization aspects to show the potential problems and shortcomings of IoT technologies [21].

As application examples from the turbomachinery era show, digital twin is a technology with great potential for improving the performance of the whole lifecycle, including design, experiment, manufacturing, assembly, operation, maintenance, and recycling phase [22]. This study shows the importance of linking other IoT technologies to digital twin. The improvement of O&M performance can be based on this integration; as an example, for architecture, O&M management shows the potentials in the integration of building information modeling (BIM), distributed sensor networks, augmented reality (AR), and digital twin [23]. Digital twin solutions can be used not only in production and service environments, but they have also potentials at the macro-level [24], where a complex system of architectures and infrastructures are integrated, as shown in the case of a GeoTwin project, where geodetic, geometric aspects of creating semantic geo-objects for a digital twin are proposed to create a digital twin city [25].

The performance of digital-twin-enabled solutions is influenced by data, information, and knowledge, available throughout the lifecycle of production and service processes. The combination of data analysis, expert knowledge, and knowledge-based systems has a great impact on this performance [26]. The maintenance practice can be influenced in many ways using digital-twin-enabled operation control. As a camera-less remote surveillance system on heavy equipment shows, IoT concepts can be used to estimate the fatigue life of machines as part of the predictive maintenance policy optimization [27].

The energy sector has focused on the application of digital-twin-enabled technologies to improve the performance of reliability analysis and related maintenance, reduce operation cost, and prevent unexpected failures [28,29]. Asset lifecycle optimization is generally focusing on both efficiency and resource optimization. Knowledge-based digital twins enable the improvement of reliability, availability, maintenance, safety, and sustainability, as the use of an RAMS² toolbox demonstrates [30].

As a probabilistic multiscale model based on digital-twin-driven framework for nondeterministic fatigue life prediction of steel bridges shows, digital-twin-enabled technologies can be used both in deterministic and uncertain environments [31]. As the European Commission defines, Industry 5.0 complements the existing Industry 4.0 approach by specifically placing research and innovation at the service of the transition to a sustainable, human-centric industry [32]. This Industry 5.0 paradigm includes human–cyber-physical systems (HCPS), where not only technical, but also human resources can be linked to IoT technologies, including digital twin solutions. The human resources represent increased complexity. The data collection, analyses, and decision-making in this complex environment require artificial intelligence and machine learning to elevate human intelligence to a supervisory level in digital-twin-enabled O&M processes [33].

The digital-twin-enabled solutions can be applied for complex infrastructure networks (production, infrastructure), but it is also possible to use them as edge systems on microcontroller-based embedded platforms. The main research fields of microcontroller-based local digital twin solutions include both the industrial and service sector, to offer solutions for flexible and low-cost industrial digitalization platforms for condition monitoring and maintenance applications, focusing on predictive maintenance processes [34]. Not only large companies, but also small and medium-sized enterprises (SMEs) can transform their conventional manufacturing system into a cyber-physical manufacturing system. This transformation is usually based on IoT technologies and uses the advances of digital twin solutions. In the case of SMEs, the application of IoT technologies and their efficient integration into business processes is difficult, because the required interface standards and data schema do not exist. An interesting direction of digital-twin-related research fields is the development of suitable data schema for the implementation of digital twin solutions to build a digital manufacturing environment [35]. Cybersecurity plays an important role in cyber-physical systems, defending computers, computer networks, mobile devices, electronic systems, and data from different types of attacks. These IoT-related cybersecurity aspects are especially important in the energy sector, where the implementation of secure embedded intelligence and integrated state awareness can lead to smart and flexible operations, optimized predictive maintenance, inherent digital security, and autonomous controls [36].

The transformation of conventional systems into cyber-physical systems leads to an increased complexity. This increased complexity has a great impact on the design and operation of these systems, where cyber-physical systems must be evaluated from a wide range of requirements’ point of view. The heterogeneous requirements focusing on time-, capacity-, service level-, and energy-efficiency-related constraints need new complex methods to define these systems requirements. Using Formal Requirements Modeling Language and the Modelica language, it is possible to formally define these requirements and verify them through simulation techniques [37,38].

The articles that addressed the maintenance policy optimization using digital twin solutions focused on availability and costs, but few of the articles aimed to provide a real-time optimization approach within a cyber-physical environment while considering energy consumption of both operation and maintenance processes. The energy efficiency influences not only the cost efficiency of manufacturing systems, but it also has a great impact on the sustainability through energy consumption reduction and greenhouse gas emission reduction.

Therefore, real-time optimization of maintenance policies in cyber-physical manufacturing systems considering energy efficiency and environmental impact still needs more attention and research. The main contributions of this article include (1) a functional model of digital-twin-enabled real-time maintenance policy optimization including ERP and MES data, and real-time failure data and status information from the digital twin; (2) a mathematical model integrating Markov decision model and value determination equations, which minimizes the energy consumption and environmental impact of O&M; (3) an evolutionary optimization tool to find the best real-time maintenance policy in digital-twin-supported manufacturing systems; and (4) a scenario analysis of real-time maintenance policy optimization.

3. Materials and Methods

The conception of digital-twin-enabled real-time maintenance policy optimization is based on both a functional model and a mathematical model. The functional model defines the technological framework, while the mathematical model includes the integration of Markov decision process and value determination equations and describes the heuristic optimization algorithm.

3.1. Functional Model of Digital-Twin-Based Real-Time Maintenance Policy Optimization

In cyber-physical manufacturing systems, the physical layer represents the technological and logistics processes of the manufacturing. The transformation of physical processes into digital processes is enabled by intelligent tools, gentelligent products, where the communication-capable smart sensors with onboard diagnostics make it possible to collect real-time data from the manufacturing system (technological, logistics and human resources, products, processes, failure, and status information) and send them to the digital twin. The digital-twin-based discrete event simulation is based on the digital replica of the manufacturing system. This DT-based simulation allows extrapolation and forecasting from system, process, resources, and product point of view (see Figure 1).

However, ERP includes a wide range of modules for the manufacturing and related logistics processes (purchasing and inventory to support procurement processes, manufacturing for production planning and scheduling, customer relationship for marketing and distribution operations, human resource management for technological, logistics, and administrative processes, etc.), but the digital replica of the manufacturing system is based on both the real-time data from the physical layer and on ERP- or MES-data, including production plan, MRP1 and MRP2, and other manufacturing-related information. The maintenance policy optimization is based on the strategies defined by the ERP maintenance module including organization, refurbishing, measurement, and object management. The real-time mathematical optimization of the maintenance policy uses information from forecasting and extrapolation scenarios and takes the initial maintenance policy generated by the ERP or MES maintenance module into consideration. The digital twin layer of the functional model is responsible for the transformation of the real physical system into a digital model, where the real-time status of the system can be taken into consideration. The digital twin model acquires information through a smart sensor network, where edge computing solutions can improve the data processing.

In the case of conventional manufacturing systems, static simulation models are used, but in the case of this approach it is possible to dynamically change the model in the discrete event simulation. In the case of this dynamic model, the real-time status of the system can be taken into consideration while making process analysis, forecasting, and extrapolation.

3.2. Mathematical Model of Digital-Twin-Based Real-Time Maintenance Policy Optimization

The model and solution algorithm of the maintenance policy optimization can be divided into two main parts. The first part is optimization of the initial static maintenance policy using Markov decision process, Howard’s policy iteration technique, and value determination equations, while the second part is the real-time policy optimization using evolutionary heuristics and value determination equation (see Figure 2).

The mathematical model and the optimization process include the following main steps in the optimization of the static initial maintenance policy: (1) definition of states of resources and potential maintenance operations; (2) definition of Markov decision process including probability transition kernel and transition matrix; (3) definition of discounted profit-based objective function, based on value determination equations; (4) definition of constraints regarding energy, time, emission, and capacity; (5) computation of optimal initial static maintenance policy using Howard’s policy iteration technique; and (6) evaluation of static initial maintenance policy from discounted profit, energy consumption, and emission point of view.

The behavior of manufacturing and logistics resources in a manufacturing system can be described with a wide range of random processes: Gaussian, Poisson, autoregressive, or Markov processes. In the case of discrete time stochastic processes, the most suitable way to describe time-dependent behavior is the Markov process, which includes a discrete sequence of states, each generated from a discrete finite state space. The potential states of the resources and the potential maintenance operations can be defined as discrete sets in the finite state space:

$$SR=\left(s{r}_1,s{r}_2,\cdots ,s{r}_i,\cdots s{r}_z\right)\mathrm{and}MD=\left(m{d}_1,m{d}_2,\cdots ,m{d}_j,\cdots m{d}_r\right),$$

(1)

where $SR$ is the matrix of the state of technological and logistics resources, $s{r}_i$ is the possible state I of the system or resource, $MD$ is the matrix of potential maintenance operations, and $m{d}_j$ is the potential maintenance operation j.

The Markov behavior of the resources is based on an initial probability distribution and a probability transition kernel, which characterizes the probabilistic dynamic of the changes of states, including the following constraint:

$$p\left(s{r}_f|s{r}_i,m{d}_j\right)\in \mathsf{\Lambda }\wedge p\left(s{r}_f|s{r}_i,m{d}_j\right)\le {\lambda }_{if} \vee p\left(s{r}_f|s{r}_i,m{d}_j\right)\ge {\lambda }_{if},$$

(2)

where $p\left(s{r}_f|s{r}_i,m{d}_j\right)$ is the probability of transforming the system from status i to status f with maintenance policy j, ${\lambda }_{if}$ is the transition probability from state $s{r}_i$ to state $s{r}_h$, $\mathsf{\Lambda }$ is the matrix of transition probabilities, and $\mathsf{\Lambda }=\left[{\lambda }_{if}\right].$

The objective function of the maintenance policy optimization problem is the discounted profit, which is based on value determination equations [3], and it is influenced by a wide range of factors, including the energy consumption of O&M:

$$\mathsf{{\rm Y}}\left(s{r}_i\right)={\zeta }_{s{r}_i,m{d}_j(s{r}_i)}+\vartheta ·{{\displaystyle \sum }}_{j=1}^{r}p\left(s{r}_f|s{r}_i,m{d}_j(s{r}_i)\right)\mathsf{{\rm Y}}\left(s{r}_f\right),$$

(3)

where $\mathsf{{\rm Y}}\left(s{r}_i\right)$ is the expected discounted profit in the case of system status i, ${\zeta }_{s{r}_i,m{d}_j(s{r}_i)}$ is the expected profit caused by maintenance policy j in the case of state of the system i, and $\vartheta$ is the discounting parameter, where $0<\vartheta <1$.

The Howard’s policy iteration technique uses the computation of a special $\mathsf{\Theta }$ parameter:

$${\mathsf{\Theta }}_D^{}\left(s{r}_i\right)=\underset{m{d}_j\in MD(s{r}_i)}{\max }\left({\zeta }_{s{r}_i,m{d}_j(s{r}_i)}+\vartheta ·{{\displaystyle \sum }}_{j=1}^{r}p\left(s{r}_f|s{r}_i,m{d}_j(s{r}_i)\right)\mathsf{{\rm Y}}\left(s{r}_f\right)\right),$$

(4)

where ${\mathsf{\Theta }}_D^{}\left(s{r}_i\right)$ is a special parameter of Howard’s policy iteration technique and it depends on the possible state i of the system or resource.

Based on the value of this $\mathsf{\Theta }$ parameter, we can find the optimal maintenance policy using the following decision:

$$\forall j:{\mathsf{\Theta }}_D^{}\left(s{r}_i\right)=\mathsf{{\rm Y}}\left(s{r}_i\right)\to D optimal decision,$$

(5)

otherwise, new iteration steps are required.

We can integrate the energy consumption into the discounted profit-based objective function, because the initial profit of the manufacturing system can be reduced with the energy consumption costs of O&M:

$${\zeta }_{i,j}={\zeta }^I-{\zeta }_i^{OPER}-{\zeta }_j^{MAINT},$$

(6)

where ${\zeta }_{i,j}$ is the income generated by the analyzed manufacturing system depending on the state of the system and the chosen maintenance policy, ${\zeta }^I$ is the initial income of the manufacturing system, ${\zeta }_i^{OPER}$ is the cost caused by the energy consumption of the operation in the case of state i of the manufacturing system, and ${\zeta }_j^{MAINT}$ is the cost caused by the energy consumption of the chosen maintenance policy j.

The costs of energy consumption of O&M can be computed using the levelized cost of energy:

$$LCoE=\frac{{{\displaystyle \sum }}_{\tau =1}^{{\tau }^{*}}\frac{I{E}_{\tau }+OM{E}_{\tau }+EG{R}_{\tau }}{{\left(1+\vartheta \right)}^{\tau }}}{{{\displaystyle \sum }}_{\tau =1}^{{\tau }^{*}}\frac{G{E}_{\tau }}{{\left(1+\vartheta \right)}^{\tau }}},$$

(7)

where $LCoE$ is the levelized cost of energy, $I{E}_{\tau }$ is the investment expenditures in the time frame $\tau$, $OM{E}_{\tau }$ is the O&M expenditures in the time frame $\tau$, $EG{R}_{\tau }$ is the energy generated in the time frame $\tau$, and ${\tau }^{*}$ is the expected lifetime of the energy generation system.

The second phase of the optimization algorithm is the real-time optimization. If the parameters of O&M are changed, the initial static maintenance policy must be revised, and a new maintenance policy must be defined using real-time data and forecasting information from situation analysis of digital-twin-enabled discrete event simulation. The real-time optimization of the maintenance policy optimization includes the following main steps: (1) data collection from digital twin, results of situation analysis of discrete event simulation, ERP and MES; (2) definition of parameters for the evolutionary optimization algorithm; (3) definition of state space; (4) computation of the new maintenance policy based on real-time data; (5) evaluation of real-time maintenance policy from discounted profit, energy consumption, and emission point of view. Within the frame of the next section are the computational results of the described real-time maintenance policy optimization problem, focusing on energy consumption and environmental impact of O&M.

4. Results

If the maintenance strategy is predefined for all subsystems of the manufacturing system, the initial maintenance strategy can be defined using Howard’s policy iteration technique. If there is any change in the parameters used to define the initial maintenance policy, the initial maintenance policy can no longer be considered optimal in the changed manufacturing environment, so it is advisable to change the strategy in real time, simultaneous with the changes in the system parameters. Within the frame of this scenario analysis, the computational results of a segmented manufacturing system are discussed from the real maintenance policy optimization point of view.

4.1. Description of the Scenario, Input Parameters of the Optimization

Within the frame of this scenario, five different levels of maintenance can be performed in the manufacturing system. There is a zero-level maintenance, which means that no maintenance is performed, while level 1 to 4 maintenances can increase the status (quality) of the manufacturing system with 1 to 4 levels. For example, a level 2 maintenance increases the status level from 4 to 2. The profit within one cycle can be defined depending on the state of the manufacturing system and the performed maintenance operation (see

Table 2. Total income depending on the maintenance level and operation profit.

Maintenance			Manufacturing System
		Level	1	2	3	4	5	6	7
Level	Cost	Profit	2000	1500	1000	800	600	400	200
0	0	-	2000	1500	1000	800	600	400	200
1	500	-		2000	1500	1000	800	600	400
2	650	-			2000	1500	1000	800	600
3	750	-				2000	1500	1000	800
4	825	-					2000	1500	1000

and

Table 3. Total profit depending on the maintenance cost and operation profit.

Maintenance			Manufacturing System
		Level	1	2	3	4	5	6	7
Level	Cost	Profit	2000	1500	1000	800	600	400	200
0	0	-	2000	1500	1000	800	600	400	200
1	500	-		1500	1000	500	300	100	−300
2	650	-			1350	650	350	150	−50
3	750	-				1250	750	250	50
4	825	-					1175	675	175

). For example, in the case of a level 5 manufacturing system, the initial income is EUR 600. Performing a level 3 maintenance, which costs EUR 750, the manufacturing system will be transformed into a level 2 manufacturing system, which has an income of EUR 1500, and the total profit can be calculated for this level 2 manufacturing system as the difference of the income of the level 2 manufacturing system and the cost of the performed level 3 maintenance.

The transition probabilities are given; they have for both parts of the manufacturing system the same values (see

Table 4. Transition probabilities of the states in the manufacturing system.

	States of the Resources in the Manufacturing System
From-To	1	2	3	4	5	6	7
1	0.75	0.22	0.03	0	0	0	0
2	0	0.65	0.27	0.04	0.04	0	0
3	0	0	0.81	0.11	0.08	0	0
4	0	0	0	0.64	0.24	0.12	0
5	0	0	0	0	0.74	0.22	0.04
6	0	0	0	0	0	0.8	0.2
7	0	0	0	0	0	0	1

). The transition probabilities define the probability of transition between two states of the manufacturing system caused by various maintenance operations.

The discounting factor is 0.87, the levelized energy cost is in the case of natural gas USD 0.062/kWh, in the case of nuclear USD 0.163/kWh, in the case of thermal solar USD 0.14/kWh, and in the case of coal USD 0.115/kWh. The emission of greenhouse gases is also given, depending on the energy generation source (see

Table 5. Specific emission rates of GHGs depending on the energy generation source in g/kWh.

Energy Generation Source	GHGs
	CO2	SO2	CO	HC	NOx	PM
Natural gas	499	0.016	0.418	0.228	2.226	0.019
Nuclear	29	0.0001	0.024	0.013	0.132	0.001
Thermal solar	85	0.002	0.073	0.04	0.396	0.003
Coal	888	0.028	0.733	0.4	3.96	0.03

).

4.2. Computation of the Initial Maintenance Policy Using Howard’s Policy Iteration Technique in a Markov Decision Process

Using Howard’s policy iteration technique, we can define the initial optimal maintenance policy from cost and energy points of view. The optimal assignment of states and maintenance levels can be defined as a set of the maintenance policies:

$$X=\left(x_1=0,x_2=1,x_3=2,x_4=3,x_5=4,x_6=4,x_7=4\right).$$

(8)

The main impacts of the initial maintenance policy on the energy-related cost and energy consumption depending on the energy generation source are shown in part A of Table 5.

4.3. Results of the Real-Time Heuristic Optimization of Maintenance Policy

Minor changes in the manufacturing system can lead to enormous profit reduction and increased energy consumption and emission. After minor/major changes in the states of the manufacturing system, the static initial maintenance policy has to be changed, because in the case of the changed manufacturing environment and system parameters the initial maintenance policy is no longer the optimal solution, and this fact can lead to increased energy consumption, energy-related cost, and increased environmental impact caused by increased GHG emission. Part B of

Table 6. Optimal initial maintenance policy using Howard’s policy iteration technique in the Markov decision process.

Optimal Maintenance Level	Status of the Manufacturing System
	1	2	3	4	5	6	7
A. Initial policy	0	1	2	3	4	4	4
Energy costs (USD)	482	982	1132	1232	1307	2184	2781
Energy consumption (kWh)
Natural gas	7774	15,839	18,258	19,871	21,081	35,235	44,864
Nuclear	2957	6025	6945	7559	8019	13,403	17,065
Thermal solar	3443	7015	8086	8800	9336	15,604	19,869
Coal	4192	8539	9844	10,713	11,366	18,997	24,188
B. Changed states with initial policy
Energy costs (USD)	482	982	1132	1232	4042	5015	5707
Energy consumption (kWh)
Natural gas	7775	15,839	18,259	19,872	65,194	80,884	92,048
Nuclear	2957	6025	6945	7559	24,798	30,766	35,012
Thermal solar	3443	7015	8086	8800	28,872	35,820	40,764
Coal	4192	8539	9844	10,713	35,148	43,607	49,626
C. Real-time optimized policy	0	1	2	3	3	3	3
Energy costs (USD)	482	982	1132	1232	2139	2764	6131
Energy consumption (kWh)
Natural gas	7775	15,839	18,259	19,872	34,493	44,589	98,889
Nuclear	2957	6025	6945	7559	13,120	16,960	37,614
Thermal solar	3443	7015	8086	8800	15,275	19,746	43,794
Coal	4192	8539	9844	10,713	18,596	24,039	53,314

shows this effect in the case of this scenario. Using the proposed real-time optimization approach, it is possible to change the maintenance policy, as part C of Table 6 shows. In this scenario, the initial maintenance policy can be optimized using the real-time data available from the digital twin and the forecasting information from digital-twin-based discrete event simulation.

As Table 6 summarizes, the optimized real-time maintenance policy led to decreased energy consumption, energy cost, and GHG emission of about 21%. The value of this reduction depends on the order of magnitude and structure of the system parameters.

Table 7. Emission reduction of GHGs in kg, as a result of the real-time optimized maintenance policy.

Energy Generation Source	GHGs
	CO2	SO2	CO	HC	NOx	PM
Natural gas	30,017.420	0.001	0.025	0.014	0.134	0.001
Nuclear	663.552	0.002	0.549	0.297	3.020	0.023
Thermal solar	2264.412	0.053	1.945	1.066	10.549	0.080
Coal	28,799.147	0.908	23.772	12.973	128.429	0.973

summarizes the impact of the real-time maintenance policy optimization on the GHG emission.

To summarize, the proposed real-time maintenance policy optimization, including static initial maintenance policy definition, and its improvement based on real-time data from digital twin makes it possible to improve the impact of maintenance policies on O&M, focusing on the energy efficiency and environmental impact of operation and maintenance. For the optimization of the maintenance policy, we used Excel Solver.

As the findings of the literature review show, the articles that addressed the maintenance policy optimization focused on costs, but only a few of them aimed to identify the challenges of real-time maintenance policy optimization from energy and GHG emission points of view. The comparison of results of the proposed methodology with those from other studies shows that the digital-twin-enabled real-time maintenance policy optimization still needs more attention and research. The reason for this is that in the case of a dynamically changing manufacturing environment, the maintenance policy must be permanently analyzed, and, depending on its permanent impact on KPIs, real-time modified and optimized. The proposed model and algorithm can obtain different objectives for time, cost, energy, emission, capacity, and various time- and capacity-related constraints of the manufacturing system.

5. Conclusions and Discussions

A methodological framework and a mathematical model were proposed that make it possible to optimize the maintenance policies in cyber-physical manufacturing environments. More generally, this paper focused on the mathematical description of the framework of real-time maintenance policy optimization using Markov decision process, Howard’s policy iteration technique, value determination equations, and evolutionary heuristics, and showed the impact of optimization on the energy efficiency and GHG emission. The main findings of the article are the following: (1) maintenance policies must be permanently revised, because in dynamically changing manufacturing environments the static policies can influence KPIs and reduce efficiency, availability, profit, and sustainability; (2) the application of Industry 4.0 technologies and the transformation of conventional manufacturing systems into cyber-physical manufacturing environments offers new solutions for real-time decision-making; (3) the integration of Markov decision process, Howard’s policy iteration technique, the value determination equations, and evolutionary heuristics supports the digital-twin-enabled optimization of real-time maintenance policy optimization; (4) the proposed real-time maintenance policy optimization caused an energy consumption and GHG emission reduction of 21%, depending on the energy generation source. The results can be generalized, because the proposed framework and methodology can be applied for maintenance optimization in production, services, infrastructure, and architecture. The proposal makes it possible to support managerial decisions; the decision making process regarding manufacturing, logistics, and maintenance can be influenced by the results of the contribution. The described methodology makes it possible to analyze the financial impact of maintenance strategies on the performance of the manufacturing system and their costs. The managerial impacts include not only the field of maintenance management, but human resource management and inventory management are also influenced by the maintenance strategy.

However, there are also limitations of the study. The manufacturing systems are segmented in many cases. This segmentation is not taken into consideration. This limitation shows a potential direction for further research. In further studies, the model can be extended to a more complex model for segmented manufacturing systems and innovative business models [39]. Second, this study only considered evolution heuristics for the solution of the real-time phase of maintenance optimization, but other heuristics and metaheuristics can also be taken into consideration. The third potential research direction is the modeling and optimization of maintenance models using fuzzy models, where the uncertainties of the manufacturing environment can be taken into consideration. The fourth potential future research direction is the combination of the above-described maintenance strategy optimization and the additive manufacturing [40], because additive manufacturing can significantly decrease the required inventory level.