Proactive Maintenance Model Using Reinforcement Learning Algorithm in Rubber Industry

Abstract

This paper presents an investigation into the enhancement of availability of a curing machine deployed in the rubber industry, located in Tamilnadu in India. Machine maintenance is a major task in the rubber industry, due to the demand for product. Critical component identification in curing machines is necessary to prevent rapid failure followed by subsequent repairs that extend curing machine downtime. A reward in the Reinforcement Learning Algorithm (RLA) prevents frequent downtime by improving the availability of the curing machine at time when unscheduled long-term maintenance would interfere with operation, due to the occurrence of unspecified failure to a critical component. Over time, depreciation and degradation of components in a machine are unavoidable, as is shown in the present investigation through intelligent assessment of the lifespan of components. So far, no effective methodology has been implemented in a real-time maintenance environment. RLAs seem to be a more effective application when it is based on intelligent assessment, which encompasses the failure and repair rate used to calculate availability in an automated environment. Training of RLAs is performed to evaluate overall equipment efficiency (OEE) in terms of availability. The availability of a curing machine in the form of state probability is modeled in first-order differential-difference equations. RLAs maximize the rate of availability of the machine. Preventive maintenance (PM) rate for four modules of 16 curing machines is expressed in a transition diagram, using transition rate. Transition rate indicates the degree of PM and unplanned maintenance rates that defines the total availability of the four modules. OEE is expressed in terms of the availability of curing machines, which is related to performance and quality. The results obtained by RLA are promising regarding short-term and long-term efficiencies of OEE, which are 95.19% and 83.37%, respectively.

1. Introduction

The manufacturing industry periodically reviews reliability and availability standards in the pursuit of continuous improvement in quality and production. The quality of a product depends on several factors, such as manufacturing processes, machine durability, and human handling skills. Production improvement is a major objective in the manufacturing industry; it is to be obtained by ensuring the availability and maintainability of machines and equipment. Therefore, reliability, availability, and maintainability (RAM) are the major factors for quality in industrial evaluation, as well as product improvement [1].

Maintenance is an activity carried out on a machine that ensures reliability and availability. It safeguards the machine from failure when the working principles of the machine have deviated from the activities specified by the machine manufacturer. Maintenance is performed in two ways, by preventing failure or by repairing the machine when failure has occurred. Continuous and periodic maintenance prevents failure; however, machines in industry are required to work continuously, meaning that faults may occur on these machines at any time. Faults cause failure. Maintenance must be carried out at specific intervals, during which the machine goes to a downstate regardless of whether it is working or not. This maintenance is known as preventive maintenance.

Failure in machine leads to downtime, stopping production. During the downtime, maintenance is carried out to remove the failures, where either a new machine replaces the failed machine, or the failed machine is serviced and restored to a working state. Whenever the machine goes into a failure state, maintenance activity called corrective maintenance is carried out to repair the machine. If a machine is working properly for a prolonged time, no maintenance activities are required. If the machine at a production site is stopped for maintenance activities, production is affected during that time. Proactive maintenance is effective because it does not stop production. A standby machine takes over production during proactive maintenance. The cost paid for these two maintenance activities are different. The amount of downtime, spare parts for the machine, and the expense of servicing determine the cost of maintenance. The Markov model [2] is a well-known mathematical tool to express availability and maintainability. The lifespan of the machine is denoted as an interval of time, and the Markov process defines the machine’s finite states within this interval of time. The Markov process [3] can be enhanced as a continuous-time Markov chain for the identification of critical components in a multi-state repairable system with N components. The identification of critical components in the machine is achieved through failure and repair rates of components in the corresponding machine state. A Markov process-implemented repairable system has spatial dependence [4]. Spatial dependence is found in a repairable system where components are arranged as a star or lattice connection, where the central component N is connected to the peripheral components N-1. Failure found in any of the peripheral components will affect neighboring peripheral components. The availability of peripheral components is determined by failure rate and the spatial pattern of the arrangement.

The study in [5] proposes an optimal maintenance policy that evaluates the steady-state probability of failure and repair after examining the Markov decision process. Effective preventive maintenance leads to minimal repair requirement after random failures. A parallel repairable system [6] consists of N identical and independent subsystems with different failure rates. First-order linear equations are solved by finding the failures and repairs in this system. A semi-Markov process finds the steady-state probability, and then condition-based maintenance (CBM) [7] is followed to obtain the steady-state availability for mechanical systems. Sensors and data-capturing technology can be installed in mechanical systems to capture their conditions.

Overall equipment efficiency (OEE) is denoted by effectiveness in achieving optimal performance in machines used in production and then the following key parameters are evaluated: machine availability, planned maintenance, and unplanned maintenance. The significance of the product is measured with respect to the overall quantity of products made on the production line, and quality is calculated by the overall number of products yielding that are defect-free. Machine-Learning Algorithms (MLAs) [8] identify machine effectiveness based on time, and MLAs are now making decisions to improve effectiveness. Data such as sensor output and machine readings from the production line are used to train MLAs. In the next level of MLAs [9], hybrid Internet of Things (IoT) will enable cyber-systems to collect data across manufacturing industries. RLAs are a subset of MLAs established to analyze the reliability of the production industry [10]. RLAs are formulated for reliability analysis by training with a dataset from the production industry and then learning the performance after execution of this dataset. Reliability analysis in industry entails checking the availability of machines where several metrics are implemented via an RLA [10] to improve reliability. High reliability improvements [11] of machines has reduced the time required for scheduled maintenance when reinforcement learning (RL) must be used. In long-term maintenance, rewards generated by the sequential actions of a machine is described as decision-based. Decision-based rewards improve the reliability of CBM for a machine. Each product on the production line of a machine gives feedback about its quality and performance. Quality and performance is derived from the availability of the machine, and feedback is a reward that indicates profit or loss. Using this reward could improve future products, and maintenance activity will be carried out if the reward indicates losses at a higher level. IoT with cyber-systems have emerged in Industry 4.0 with the handling of big data, which defines OEE in manufacturing industries. OEE calculation automation [12] prepares software and hardware models where hardware collects the start-, stop-, and downtime of machines on a production site, and gathers product quantity data for each working time or shift. Here, downtime represents failure and repair data, and then availability is derived using these data. Availability and maintainability are calculated in the development of software using Visual Basic, which is embedded in an ATmega 165 microcontroller. Generally, OEE is assessed through the availability of a machine and its performance and quality. Availability was affected due to six major stoppages in the ice-cream industry [13] which were failure, adjustment time, defective processes, reduced performance, temporary malfunction, and end of component life span. These stoppages were solved by total productive maintenance (TPM) that uses underlying data collected from the automated production system of the ice-cream industry.

Manufacturing industries are product industries that may produce metals, alloys, chemicals, automobiles, cement, plastics, oil and gas, food and animal feed, rubber products, and so on. The total quality management concept improves RAM principles in the above-mentioned industries. Research presented in this paper shows how RAM principles can influence quality and performance improvement in the rubber industry. Several existing studies are considered in this literature review below.

The oil and gas industry [1] has evaluated RAM principles to improve performance using three-out-of-four systems. In three-out-of-four systems, three units are operating, and one unit is in standby mode. This system fails when two units fail, and its behavior is modeled by the Markovian approach. Differential equations are used to define system availability as steady state; then, furthermore, they are solved using Laplace transform in a transient state to obtain reliability. Finally, this system model can be applied to the real-time data taken from the oil and gas industry to measure performance.

The study in [14] demonstrated the improvement to maintenance that increases system availability using Stochastic Petri Nets (SPN). This consists of different classes with two transitions—one for equipment failure, and one for equipment repaired. MINITAB data-processing software modeled the preventive maintenance plan using SPN. The oil and gas industry [15] has implemented management strategies to improve productivity. It has used common techniques of mean time to failure, downtime of equipment, and system availability on a reciprocating compressor unit. A reciprocating compressor unit is a piece of sophisticated equipment in this industry, and research in [15] assessed the availability of this equipment in a cost-effective manner. Its main function is to export gas into a manifold system that satisfies the operational and maintenance requirements of subsystems of compressors, such as cylinders, piping, inter-stage vessels, crankcases, and electric motors or gas turbines.

Availability and assessment analysis of multiple systems have been proposed for the power-plant industry [16]. Boiler air circulation is the major system in this industry and the Markov birth–death process has been developed for availability analysis. A transition diagram represents the boiler air circulation system states of reduced capacity, full working, and failure. Probabilistic differential equations were solved with the help of transition diagrams to find the system’s steady-state probability. Ship machinery performance [17] was evaluated to assess RAM parameters using a continuous Markov process. The main problem in the shipping industry is formulated and represented as a multi-variable and multi-constraint engineering problem. An engine cooling system is a major piece of equipment on a ship that requires optimization from the simulation of RAM parameters.

The availability of chemical processing plants [18] was determined not only by subsystems in production, but also by storage unit. Since multi-production lines in this plant have integrated storage units, an analytical model of availability analysis for integrated storage units was given in this work [6]. A semi-Markov process with regenerative states was proposed to capture the character of the storage unit. A Load–Haul Dumper (LHD) [19] was used for underground mining with large datasets. Three-parameter Weibull distribution analyzed these datasets for reliability estimation in a statistical manner in an LHD. Preventive maintenance intervals were identified from reliability estimation for subsystems of an LHD with shovel dumpers, draglines, and in-pit crushing. Data collection consisted of failure data and a probability density function (PDF) that was collected from the subsystems of the LHD over a particular period of time.

Certain reviews of reliability analysis have been performed on tunneling equipment [20], which consists of the Markov process, statistical analysis, fault tree methods, and failure mode and effect analysis (FMEA). Failure and repair data were regularly collected from the equipment. With these data implemented in the above-mentioned analysis, availability and maintainability schedules were prepared for tunneling equipment subsystems. The output of a multi-stage engine [21] was modeled in a discrete state-continuous Markov process. Using observed data for the output of the multi-stage engine, a multi-state Markov model was established. Output power and engine capacity were calculated from the multi-state Markov process within the time domain, and calculated data were used to forecast engine reliability.

RAM analysis has helped the ice-cream industry [22] assess preventive maintenance intervals for planning and organizing effective maintenance schemes. Failure data were collected from RAM analysis over a period of 12 months. An ice-cream machine is a network of subsystems doing pasteurization, homogenization, aging, freezing, and packaging. According to these operations, failure data were collected. As with the ice-cream industry, the sugar industry [23] has a feeding system that comprises subsystems for cutting, crushing, carrying, and heat-generation operations. The Markov process analyzed the reliability and availability of the subsystems, and states of the subsystems were modeled using Markov transition diagrams. The feeding system in the sugar industry has states of work, reduced work, and failure. In a Markov transition diagram and its state, a genetic algorithm is executed to find and optimize steady-state availability.

A machine-learning survey [24] was applied to a factory floor production line for the prevention of equipment failures, and then performance of predictive maintenance was measured. Mixed-Integer Linear Programming (MILP) metaheuristics [25] were used to optimize the maintenance planning in nuclear power plants. MILP dealt with the maintenance planning of medium and large nuclear power plants. Periodic preventive maintenance policies [26] have been considered for a depreciating reparation system that overhauls a system during planned preventive maintenance. It improved reliability performance. A Non-Homogeneous Poisson Process (NHPP) was applied to industry for the repair of components in minimal cases. Computerized maintenance management (CMMS) [27] was considered to improve decision-marking regarding maintenance excellence. Failures were frequently updated using CMMS, which optimizes equipment reliability as per the failure-mode effective-analysis method.

The recent trend for assessing OEE evaluates machine learning in supervised, unsupervised, and reinforcement learning. Reinforcement learning is more suitable for OEE assessment than supervised or unsupervised learning. Section 2 debates RLA use in maintenance. In Section 3, system modeling is done, and RLAs show improvement in maintenance. Conclusions are drawn in Section 4.

2. System Description and RL-Based Intelligent Diagnosis

RAM analysis in the rubber industry using vulcanized or sometimes vulcanized rubber is done using a curing machine, where high-pressure steam is used to cure the rubber in the vulcanizer. The process of vulcanizing using steam is called curing. A curing machine is an indispensable system in the rubber industry. It is arranged as four independent machines, and each machine has four pots. Curing setups in all pots are identical, consisting of the following subsystems or components: bottom vessel, top lid, top lid closure, bottom lid closure, vacuum pump unit, gasket, low/high pressure sensors, and low/high temperature sensors.

A detailed investigation of RAM analysis for the curing system has been established in the rubber industry. Several advantages have been introduced by RAM analysis to improve performance. Performance increases productivity on the curing production line.

(a)

Criticality levels have been introduced in RAM analysis to identify critical levels in curing subsystems.

(b)

Identification of critical levels in subsystem helps provide optimal availability and maintainability in curing subsystems.

(c)

Criticality levels define the CBM; it is done in this investigation via industrial IoT-connected sensors.

(d)

Unplanned maintenance over the long term is made to identify unexpected failures, and maintenance time is derived to predict availability.

(e)

To find failure and repair data, curing machine pots are modeled by a Continuous-Time Markov Process (CTMP).

(f)

The RLA is trained using a CTMP to analyze the availability of the curing machine.

(g)

The CTMP is solved to obtain the steady-state availability of all modules of the curing machine of first-order differential-difference equations.

(h)

Using steady-state availability, performance, and quality, the OEE of the curing machine is evaluated.

2.1. Criticality Level Analysis in a Curing Machine

The main objective of the investigation presented in this paper is to analyze the criticality of the machine. The criticality of the machine is introduced in [28], and quantifies the manufacturing plant performance. An estimation of performance uses certain methodologies and tools that estimate the cost of production, including the cost involved in machines, measuring of the cost of reliability and maintenance of these machines, and finally operational and labor costs. Among these costs, a particular cost could improve the production and lifespan of a machine drastically. Contrasting the cost of production improvement with the cost of failure and maintenance, a particular subsystem or machine in a plant may have a higher failure rate and require a higher degree of maintenance. This subsystem is known as a critical component, which leads to the shutdown of production due to higher failure compared with the failure rate of the remaining components in the plant. Therefore, a critical level starts at a higher component failure rate and has a lower failure rate for the remaining components.

In the proposed investigation, the critical level includes operation and maintenance that means a workforce is required for the manufacturing plant to operate. The workforce comprises management personnel, engineers, technicians, labor, and so on. The cost paid to the workforce may be higher than the cost of machines and their maintenance. Critical levels are sought for curing machines in the rubber industry from all aspects of the workforce, machines, subsystems, and RAM analyses. Generally, the critical levels are the four levels assigned to the components of the curing machine in the proposed investigation. Level 1 is environmental safety of the workforce, where loss of life or permanent disability may be found when failure has occurred in any subsystem of the plant [29]. This particular subsystem is said to be the most critical subsystem in the rubber industry.

Environmental safety has a higher failure rate and requires frequent maintenance. Some subsystem failure severely affects the production line. It does not cause problems in safety; however, failure in a subsystem shuts down the remaining subsystems of the plant. Failure in this subsystem is critical for stopping overall production, and this is noted as Level 2 criticality. The cost of repairs for failed components and the cost of the workforce to remove failures influences profits and the durability of the machines. The cost of failures could worsen the industry financially, and this is Level 3. Apart from safety, production, and repairs, changes to the subsystem reduce the quality and quantity of production. Changes to subsystems do not lead to failures; nevertheless, specification of the products on the production line is affected. A lack of workforce and changes to operating conditions/working processes create errors in the subsystem at a specific level. This is Level 4 in criticality. The curing machine is shown in Figure 1.

According to the functionality of the components of the curing machine, critical levels are set. A curing machine is a pressurized container where rubber is processed at a high temperature and at a certain pressure. The curing machine has a chance of exploding at high pressure if a pressure-relief valve is not released. If the pressure of the curing machine exceeds the specified pressure for the rubber-processing stage, the pressure-relief valve must release the excess pressure, otherwise the container could explode. During the exploding state, the safety of the surrounding environment of the curing machine is very critical, and loss of the life or permanent disability for working people in surrounding environment may happen. Thus, Level 1 criticality is assigned to the pressure-relief valve for frequent failure/maintenance, and it is the most critical component in the curing machine. Periodical preventive maintenance is recommended for the pressure-relief valve.

This assures the maximum level of availability during all working periods of the curing machine. A substance may block the pressure-relief valve during the working period; this must be checked during every instance of loading and unloading rubber into the curing machine. In this instance, the pressure-relief valve will have deteriorated from its physical properties due to high temperature and pressure, which is happening in an overall working period in a step-by-step manner. Failure occurring in low and high-pressure sensors move the curing machine into an unavailable state in the production line, since the wrong pressure readings affect the quality of the processed rubber inside the curing machine. After taking the vulcanized rubber from the curing machine, it will be found defective when the quality of the vulcanized rubber is verified. In this defective state, production is stopped to check the reliability of these sensors, whether sensors are giving the exact pressure range of the container of the curing machine or not.

Backpressure is created in the container when vulcanized rubber is taken from the container, and, at the same time, unprocessed rubber is loaded into the container for vulcanizing. It is found in this maintenance situation that some pressure remains during the loading/unloading of rubber in the container. In this situation, the pressure range for the newly loaded rubber is sensed by the sensors as wrong, which is at different levels of pressure from the actual specified pressure level for newly loaded rubber. In both situations of failures to pressure sensors and backpressure maintained in the container of the curing machine, preventive maintenance is required to check the reliability of the sensors, which is performed during downtime or when production is in a stopped state. This is Level 2 criticality for stopping production.

2.2. RL-Based Intelligent Diagnosis

The arrangement of the curing machine is discussed in the system description. Here, an RL-based intelligent diagnostic is proposed to fulfil the next objective for improving the OEE of the curing machine in this investigation. Intelligent diagnostics automate the process for analyzing the status of the availability [22] of components in the curing machine. It also simplifies the planned short- and medium-term and unplanned long-term maintenance activities in modules of the curing machine. Unplanned maintenance is an overhauling operation on all components in all curing machines, and leads to undefined downtime for long-term maintenance. Undefined downtime cuts off products from the curing machine and affects the reward defined by RL. A reward may be premeditated by successful repair following short-, medium- and long-term maintenance. The machine then prepares for production. If the machine needs to be down for an indefinite time, it affects production. The reward may be profit or loss, depending on the successful repair rate by the machine. If the reward indicates a penalty, it is known as a loss that is deducted from the profit of the industry. Loss includes a cost for inspection and servicing the components of the curing machine, and if replacement components are needed, that cost is added to the loss. Failure occurs in any component of curing machine where identification of the failure is hard. Then, there is an overhaul of all components of this machine. In this state of maintenance, maintenance time is not estimated. The maintenance schedule [11] for preventive maintenance is prepared under the status of reliability. This schedule is compared with the datasheets of components in the curing machine. This depends on the state of the components, the servicing due, servicing history, and repair details. All automation, including maintenance scheduling and servicing details, are controlled through a microcontroller. This form of maintenance is applicable only for the short and medium term. Sudden failure followed by undefined downtime leads to the curing machine entering the overhaul state in an unplanned manner. The microcontroller and sensors interface using industrial IoT connectivity. The microcontroller has the facility to store datasheets of components of the curing machine, reliability and maintenance schedules, and the history of the status of components of the curing machine [10]. Present failure and repair rates of the components are modeled using first-order differential-difference equations. RL is implemented to train the Markov decision process with finite states formulated by first-order equations. Finally, steady-state probabilities for the overall performance of the curing machine are solved in Laplace transforms. Hereafter, mathematical modeling for RAM analysis is embedded in a microcontroller-comprised intelligent system when the corresponding RL is executed. Mathematical modeling of the steady-state probabilities for the curing machine regarding failure and repair rates is made using the sensors present in the curing machine. Current repair and failure rates of the components are compared to the datasheets of these components given by the respective manufacturers. This type of automation is collectively called Industry 4.0.

Sensors attached to components give the status of the curing machine temperature and pressure. During operation, values from the sensors are not stable. The unstable sensor values indicate a steady state slowly moving to a degraded state. After reaching an unstable state, the machine is still working, and the alarm in the corresponding component is sounded, which shows a warning to start maintenance activities on the component. Intelligent assessment followed by intelligent maintenance is started. This activity states whether a particular component needs to be serviced or replaced, and then maintenance history is prepared and stored in the corresponding information system. The information system is industrial IoT and cloud storage. From cloud storage for the existing maintenance history, a future maintenance plan can be forecast as a weekly, monthly, or yearly schedule. Along with the maintenance plan, the overall system efficiency in the current state is calculated. Sensor data are stored in the information system based on numbers of working shifts. These data are used to discover failure rate, repair rate, and PM rate. Unexpected maintenance rate is also calculated in intelligent maintenance system that is added to the existing maintenance history and is further remedied for this maintenance, to reduce the machine downtime. The availability of the curing machine derived from the following reinforcement learning algorithm has been obtained as per Equation (1), and iterations were made following the Q Learning matrix as shown in Figure 2.

3. System Modeling for Overall Equipment Efficiency Analysis

3.1. Reinforcement Learning in the Markov Model

In RL, Q learning uses the S states followed by Actions A generally, and it is shown in Figure 3.

Elaborated Q learning is iteratively found in the transition function T (S, A, S’). After each iteration, the reward function R (S, A, S’) is obtained as per Equation (1).

$$Q \left(S, A\right)={\displaystyle \sum }_{s^{\prime }}T \left(S, A, S^{\prime }\right)+\left[R \left(S, A, S\prime \right)+\gamma V*\left(S^{\prime }\right)\right]$$

(1)

A discount factor γ is updated through each iteration to define future rewards that denote the profit and loss or penalty. Value V can update the discount factor from the iteration of the starting state to the final state. The performance model for the curing machine is applied as per the model of Q learning in Equation (1) that evaluates OEE. In this section, Q learning is explained as follows: Actions A of failure, repair and maintenance are the functions found in these states. According to the actions defined in Q learning, the system of the curing machine gives rewards in the form of profits and loss. Applying the S states, A actions and R reward called training data, RL is formulated to make the necessary decisions for a curing machine, such as when to perform short-, medium-, and long-term maintenance. In this section, four modules with 16 pots from a curing machine are taken for availability analysis with the help of a transition diagram. Functional, degraded, and overhaul maintenance are mentioned as states S in RL, which are modeled in Markov decision-making for the system of a curing machine, as shown in Figure 4. Training data (S, A, R) is collected in the transition diagram in Figure 4 such that the rewards of profit P or loss L indicate successful repair in the short or medium term and unsuccessful repair in the long term. Unsuccessful repair in a curing machine pot transitions the functional state into the overhaul state where the system will remain as long as unknown failures are identified and repaired. Indefinite time is needed to overhaul the curing machine, and now reward R is changed into loss, sometimes known as punishment, when the production of the curing machine is stopped completely. Training data for long-term maintenance is known as (S, A, (R→L)). Successful repair in a curing machine is carried out in short/medium-term maintenance and its reward belongs to reduced profit as its production count is reduced from planned production. After carrying out short/medium maintenance in a degraded state, the system state is returned to a functional state and maintenance time is estimated. Then, production count is also affected. Training data for scheduled maintenance in the short and medium term is given as (S, A, (R→P)). Preventive maintenance is progressed in a planned manner of short- and medium-term schedules, whereas unplanned maintenance is performed in an overhauling state. State description for maintenance rate and overhaul rate are shown in Figure 4 with appropriate notations.

3.2. Mathematical Description for the Estimation of Availability in Scheduled/Unscheduled Maintenance

The period of working time for a machine or component is estimated where lifespan is an actual working time. In the actual working time, manufacture of this machine prescribes the scheduled maintenance for its effective operation. Then, maintenance takes some amount of time in the short or medium term at proper intervals obtained by the actual working time. However, the machine will experience downtime in its working period, and downtime occurs due to sudden failure. Sudden failure needs repair; consequently, failure time and repair time is included in downtime. Unexpected downtime is resolved by an overhaul state for long-term maintenance. Repair time is required for service or replacement of the machine or component. Then, the availability of the machine is estimated from the known working time and the unexpected downtime. Scheduled maintenance is executed by short- and medium-term maintenance.

$$Availability \left(Av\right)=\frac{Period of availability-\left(unscheduled\right)unexpected downtime}{Period of working time}$$

(2)

Applying the reinforcement learning algorithm on short-, medium-, and long-term maintenance, the Markov model is trained to give rewards according to failure rate, service rate, and maintenance rate of the curing machine. In short/medium-term maintenance, trained data (S, A, (R→P)) provides the availability of the curing machine. At the same time, unexpected downtime in long-term maintenance diminishes the rewards into loss. This shows the reduction of the cured product count in the production lines since its failure and repair rate cannot be determined. Therefore, availability of the curing machine is calculated by means of trained data (S, A, (R→L)) when the curing machine persists in the overhaul state represented in long-term maintenance.

Planned and unscheduled maintenance are the performance indicators for a machine or a component of a machine that determines the availability at these stages. These states are illustrated in Figure 5 with a respective probability of successful and unsuccessful repairs. The maintenance state belongs to the component that attains its maximum lifespan, which is predictable via effective maintenance and availability. Such effective activities are assessed internally with the help of the present component state and its manufacturer specification. If this maintenance is successful in a planned manner, the system returns to a functional state. In an unplanned manner with a long-term maintenance state, failure of the component is not identified, and the system goes to infinite downtime. At this point, the system is kept in an overhaul state to identify failures across all its components and repair those components to remove the failures. Even though downtime of this state depends on the identification of failures and the degree of repairs, the system retains a functional state after repairs.

The system is modeled to handle both states. The following first-order difference-differential equation is formulated with their failure and repair rates in a single module of four pots as an example.

F₀(t)_A + Z₀F₀(t)_A = ϑ₁(1-q) F₁(t) + µ₁F₂(t) + ϑ₂(1-q) F₃(t) + µ₂F₄(t) + ϑ₃(1-q) F₅(t) + µ₃F₆(t) + ϑ₄(1-q) F₇(t) + μ₄F₈(t)

(3)

The transition rate for the degraded state, as well as maintenance rate at the overhauling state, is modeled in pot 1 of module 1 in following equations:

F₁(t) + ϑ₁F₁(t) = ε₁F₀(t)

(4)

F₂(t) + µ₁F₂(t) = ϑ₁bF₁(t) + λ₁F₀(t)

(5)

Here, the transition is required to specify the degraded state when known failure is found in repairable pot 1, as denoted in Equation (3).

Equation (4) points out the unknown failures for overhauling state that accumulates past and present failures.

Similarly, the remaining three pots in the same module are given in following equations:

F₃(t) + ϑ₂F₃(t) = ε₂F₀(t)

(6)

F₄(t) + µ₂F₄(t) = ϑ₂bF₃(t) + λ₂F₀(t)

(7)

F₅(t) + ϑ₃F₅(t) = ε₃F₀(t)

(8)

F₆(t) + µ₃F₆(t) = ϑ₃bF₅(t) + λ₃F₀(t)

(9)

F₇(t) + ϑ₄F₇(t) = ε₄F₀(t)

(10)

F₈(t) + µ₄F₈(t) = ϑ₄bF₇(t) + λ₄F₀(t)

(11)

Applying the Laplace transform on Equations (2)–(10),

SF₀(s)_A + Z₀F₀(s)_A = 1 + ϑ₁ (1-q) F₁(s) + µ₁F₂(s) + ϑ₂ (1-q) F₃(s) + µ₂F₄(s) + ϑ₃ (1-q) F₅(s) +µ₃F₆(s) + ϑ₄ (1-q) F₇(s) + μ₄F₈(s)

(12)

SF₁(s) + ϑ₁F₁(s) = ε₁F₀(s)

(13)

SF₂(s) + µ₁F₂(s) = ϑ₁bF₁(s) + λ₁F₀(s)

(14)

SF₃(s) + ϑ₂F₃(s) = ε₂F₀(s)

(15)

SF₄(s) + µ₂F₄(s) = ϑ₂bF₃(s) + λ₂F₀(s)

(16)

SF₅(s) + ϑ₃F₅(s) = ε₃F₀(s)

(17)

SF₆(s) + µ₃F₆(s) = ϑ₃bF₅(s) + λ₃F₀(s)

(18)

SF₇(s) + ϑ₄F₇(s) = ε₄F₀(s)

(19)

SF₈(s) + µ₄F₈(s) = ϑ₄bF₇(s) + λ₄F₀(s)

(20)

To obtain the steady probability for availability in module 1, the lifespan of the pots in module 1 needs to be calculated between the functional state to maintenance state and functional state to overhaul states. Initially, time is unbound, and in this case

$$\frac{d}{dt}=0,$$

t —> infinity

Overall availability is calculated in all three states. Then Equation (10) is simplified as in the following equation.

Z₀F₀ = 1 + ϑ₁(1-q) F₁+ µ₁F₂ + ϑ₂(1-q) F₃ + µ₂F₄ + ϑ₃(1-q) F₅ + µ₃F₆ + ϑ₄(1-q) F₇ + μ₄F₈

(21)

The simplification of Equations (11) and (12), pot 1 repair rate, and maintenance rate are

$$F_1=\frac{{\mathsf{\epsilon }}_1}{{\vartheta }_1} F_0\left(\mathrm{s}\right)$$

(22)

$$F_2=\frac{{\mathsf{\epsilon }}_{1}q+{\lambda }_1}{{\mu }_1} F_0$$

(23)

Similarly, for pot 2, 3 and 4

$$F_3=\frac{{\mathsf{\epsilon }}_2}{{\vartheta }_2} F_0\left(\mathrm{s}\right)$$

(24)

$$F_4=\frac{{\mathsf{\epsilon }}_{2}q+{\lambda }_2}{{\mu }_2} F_0$$

(25)

$$F_5=\frac{{\mathsf{\epsilon }}_3}{{\vartheta }_3} F_0\left(\mathrm{s}\right)$$

(26)

$$F_6=\frac{{\mathsf{\epsilon }}_{3}q+{\lambda }_3}{{\mu }_3} F_0$$

(27)

$$F_7=\frac{{\mathsf{\epsilon }}_4}{{\vartheta }_4} F_0\left(\mathrm{s}\right)$$

(28)

$$F_8=\frac{{\mathsf{\epsilon }}_{4}q+{\lambda }_4}{{\mu }_4} F_0$$

(29)

Applying the normalizing condition for short- and medium-term maintenance of four modules

$${\displaystyle \sum }_{i=0}^4{F}_i=1$$

we obtain

$$F_{OA}={\left(1+{\displaystyle {{\displaystyle \sum }}_{i=1}^4}{F}_i \right)}^{-1}$$

(30)

$$F_{OA}={\left[1+F_1+F_3+F_5+F_7\right]}^{-1}$$

(31)

$$F_{OA}={\left[1+\frac{{\epsilon }_1}{{\vartheta }_1}+\frac{{\epsilon }_2}{{\vartheta }_2}+\frac{{\epsilon }_3}{{\vartheta }_3}+\frac{{\epsilon }_4}{{\vartheta }_4} \right]}^{-1}$$

(32)

Machine availability of short/medium-term maintenance for Module A is formulated as

F_0A = Av_A

Similarly, availabilities of module B, C and D are

$$F_{OB}={\left[1+\frac{{\epsilon }_5}{{\vartheta }_5}+\frac{{\epsilon }_6}{{\vartheta }_6}+\frac{{\epsilon }_7}{{\vartheta }_7}+\frac{{\epsilon }_8}{{\vartheta }_8} \right]}^{-1}=A_{VB}$$

(33)

$$F_{OC}={\left[1+\frac{{\epsilon }_9}{{\vartheta }_9}+\frac{{\epsilon }_{10}}{{\vartheta }_{10}}+\frac{{\epsilon }_{11}}{{\vartheta }_{11}}+\frac{{\epsilon }_{12}}{{\vartheta }_{12}} \right]}^{-1}=A_{VC}$$

(34)

$$F_{OD}={\left[1+\frac{{\epsilon }_{13}}{{\vartheta }_{13}}+\frac{{\epsilon }_{14}}{{\vartheta }_{14}}+\frac{{\epsilon }_{15}}{{\vartheta }_{15}}+\frac{{\epsilon }_{16}}{{\vartheta }_{16}} \right]}^{-1}=A_{VD}$$

(35)

To simplify the availability for short/medium-term maintenance, the transition and maintenance rates define the scheduled time for maintenance to find out the availability of the curing machines in each pot.

$$F_{OA}={\left[1+{{\displaystyle \sum }}_{i=1}^n\frac{{\epsilon }_1}{{\vartheta }_1}+\dots +\frac{{\epsilon }_n}{{\vartheta }_n}\right]}^{-1}$$

(36)

$$F_{OA}=\left(S, A, \left(R\to P\right)\right)={\left[1+{\displaystyle \sum }_{i=1}^n\frac{{\epsilon }_n}{{\vartheta }_n}\right]}^{-1}$$

(37)

where A is a pot name and n is the number of curing machines in module or pot A.

Here, short/medium-term maintenance rates are predicted, and these are taken into account for the measurement of availability. The reduction of production counts does not affect rewards in the form of profit when planned maintenance time is known, which includes the time for machine setup, service time, and time for replacement.

The same normalizing condition is applied to long-term maintenance.

Module A:

$$F_{OA}={\left[1+F_2+F_4+F_6+F_8\right]}^{-1}$$

(38)

$$F_{OA}={\left[1+\frac{{\epsilon }_{1}q+{\lambda }_1}{{\mu }_1}+\frac{{\epsilon }_{2}q+{\lambda }_2}{{\mu }_2}+\frac{{\epsilon }_{3}q+{\lambda }_3}{{\mu }_3}+\frac{{\epsilon }_{4}q+{\lambda }_4}{{\mu }_4}\right]}^{-1}=A_{VA}$$

(39)

Similarly,

$$F_{OB}={\left[1+\frac{{\epsilon }_{5}q+{\lambda }_5}{{\mu }_5}+\frac{{\epsilon }_{6}q+{\lambda }_6}{{\mu }_6}+\frac{{\epsilon }_{7}q+{\lambda }_7}{{\mu }_7}+\frac{{\epsilon }_{8}q+{\lambda }_8}{{\mu }_8}\right]}^{-1}=A_{VB}$$

(40)

$$F_{OC}={\left[1+\frac{{\epsilon }_{9}q+{\lambda }_9}{{\mu }_9}+\frac{{\epsilon }_{10}q+{\lambda }_{10}}{{\mu }_{10}}+\frac{{\epsilon }_{11}q+{\lambda }_{11}}{{\mu }_{11}}+\frac{{\epsilon }_{12}q+{\lambda }_{12}}{{\mu }_{12}}\right]}^{-1}=A_{VC}$$

(41)

$$F_{OD}={\left[1+\frac{{\epsilon }_{13}q+{\lambda }_{13}}{{\mu }_{13}}+\frac{{\epsilon }_{14}q+{\lambda }_{14}}{{\mu }_{14}}+\frac{{\epsilon }_{15}q+{\lambda }_{15}}{{\mu }_{15}}+\frac{{\epsilon }_{16}q+{\lambda }_{16}}{{\mu }_{16}}\right]}^{-1}=A_{VD}$$

(42)

Then, the overall machine availability of the four modules is obtained as planned, as well as long-term maintenance.

To simply the availability of long-term maintenance, transition rate, failure rate and repair rate describe downtime in unscheduled maintenance where the curing machine enters the overhaul state. Complete inspection, service and repair is handled to recover the curing machines from the overhaul state and return them to the functional state. Failure rate and repair rate are discovered for the particular module of the curing machine, then the availability of the curing machines in each pot is given in the following simplified equation.

$$F_{OA}={\left[1+{\displaystyle \sum }_{i=1}^n\frac{{\epsilon }_{1}q+{\lambda }_1}{{\mu }_1}+\dots +\frac{{\epsilon }_{n}q+{\lambda }_n}{{\mu }_n}\right]}^{-1}$$

(43)

$$F_{OA}=\left(S, A, \left(R\to P\right)\right)={\left[1+{{\displaystyle \sum }}_{i=1}^n\frac{{\epsilon }_{n}q+{\lambda }_n}{{\mu }_n}\right]}^{-1}$$

(44)

q is the probability of how long the curing machine is kept between a degraded state and overhaul state with specific failure and repair rate. This probability indicates the degree of unsuccessful repairs corrected in the overhauling state, and shows the reduction rewards in the form of production loss. In the RL model trained as a first-order difference-differential equation, q is the reward function that indicates successful repair of the degraded state moving to the function state. Reward function q pays the losses in terms of punishment when the system state moves to the overhaul state for unscheduled maintenance.

In general, the availability of the curing machine for long-term maintenance is written as an RL training model with respective states of functional, degraded and overhaul. At the same time, scheduled maintenance implements the RL model for the functional state with a known maintenance rate. Maintenance cost is deducted from the profit, whereas it highly affects profit that changes into loss in long-term maintenance. To evaluate the overall equipment efficiency of modules, further performance quality of all modules is required. OEE is evaluated for unscheduled long-term maintenance if effective availability is discovered. Availability does not concern the impact on reward for short/medium-term maintenance when downtime is scheduled, and a reduction of production count on the production line is estimated. Performance of the curing machine is measured based on the number of cured products delivered from all pots at a maximum rate at given production time

$$\mathrm{Performance} \left(P_f\right)=\frac{ ACP\times TCPP}{TPT}$$

(45)

Quality is another metric to achieve the maximum rate of defective free products from the overall products in the production site of curing machines.

$$Quality \left(Q\right)=\frac{TGP}{PMC}$$

(46)

OEE = Av × P_f × Q

(47)

Applying the availability found by RL at Equations (37) and (44), the performance found in Equation (45), and the quality found at (46), OEE can be written as per Equation (48):

$$\mathrm{OEE}=\left\{\left(S, A, \left(R\to P\right)\right), \left(S, A, \left(R\to L\right)\right)\right\}\times \frac{ ACT\times TCPP}{TPT}\times \frac{TGP}{PMC}$$

(48)

where,

(S, A, (R→P) is the availability found in planned maintenance with profit

(S, A, (R→L) is the availability found in unplanned maintenance with loss or punishment

ACP = Actual curing time of the product

TCPP = Total no of cured product produced

TPT = Total Production Time

TGP = Total no of Good Products

PMC = Planned Month Counts

MTTF = Mean Time To Failure

NR = No of repairs

TMT = Total maintenance time

Av = Availability

P_f = Performance

Q = Quality

S = State

A = Action

R = Reward

α = Learning rate

γ = Discount factor

For Q learning iteration method:

In other methods of short- and medium-term maintenance, calculating the availability of the curing machine during the initial iteration with the value of Q (S, A) is considered to be zero. Then, after the first iteration following the RL-based algorithm, the following values Q (S, A) are obtained as shown in Figure 6.

$$\mathrm{Q}\left(\mathrm{S},\mathrm{A}\right)=\left[\begin{array}{c}0.010 0.015 0.011 0.010\\ 0.000 0.000 0.000 0.000\\ 0.000 0.000 0.000 0.000\\ 0.000 0.000 0.000 0.000\end{array}\right]$$

For the 31st iteration, the following values Q (S, A)

$$\mathrm{Q}\left(\mathrm{S},\mathrm{A}\right)=\left[\begin{array}{c}0.310 0.315 0.311 0.310\\ 0.945 0.950 0.928 0.961\\ 0.476 0.455 0.452 0.435\\ 0.990 0.998 0.995 0.992\end{array}\right]$$

Calculating the overall equipment efficiency from Equation (48)

= (S₄,A₂) × 0.9717 × 0.9816
= 0.998 × 0.9717 × 0.9816
= 0.9519 × 100
= 95.19%

The same method has been followed for long-term maintenance and OEE was 83.37%.

Overall equipment efficiency obtained by the iteration for four modules.

The proposed mathematical modeling is applied to the curing machine used in the rubber industry (Fenner India Limited) in Tamilnadu, India. To verify the proposed RL modeling for its effectiveness, month data from July 2021 is shown in

Table 1. Various factors obtained from four modules.

Module	Pot	Failure Rate		Repair Rate		Transition Rate		Maintenance Rate
Module A	PA1	λ1	0.008	µ1	0.43	ε1	0.01	ϑ1	0.61
	PA2	λ2	0.009	µ2	0.51	ε2	0.009	ϑ2	0.59
	PA3	λ3	0.008	µ3	0.49	ε3	0.011	ϑ3	0.65
	PA4	λ4	0.005	µ4	0.55	ε4	0.007	ϑ4	0.62
Module B	PB1	λ5	0.011	µ5	0.51	ε5	0.008	ϑ5	0.59
	PB2	λ6	0.01	µ6	0.48	ε6	0.006	ϑ6	0.59
	PB3	λ7	0.01	µ7	0.45	ε7	0.011	ϑ7	0.63
	PB4	λ8	0.009	µ8	0.49	ε8	0.009	ϑ8	0.62
Module C	PC1	λ9	0.007	µ9	0.51	ε9	0.008	ϑ9	0.59
	PC2	λ10	0.008	µ10	0.55	ε10	0.009	ϑ10	0.63
	PC3	λ11	0.009	µ11	0.6	ε11	0.009	ϑ11	0.62
	PC4	λ12	0.007	µ12	0.59	ε12	0.01	ϑ12	0.58
Module D	PD1	λ13	0.011	µ13	0.53	ε13	0.011	ϑ13	0.63
	PD2	λ14	0.009	µ14	0.48	ε14	0.009	ϑ14	0.61
	PD3	λ15	0.009	µ15	0.49	ε15	0.008	ϑ15	0.59
	PD4	λ16	0.011	µ16	0.53	ε16	0.01	ϑ16	0.6

for failure, repair, transition, and maintenance rates, which represent the modules of curing machines to calculate availability, maintainability and performance. After obtaining the performance and quality, the overall equipment efficiency is calculated, and the data is shown in

Table 2. OEE on the curing machine for four modules.

MODULE	POT	Overall Equipment Efficiency (OEE) %
		Short- and Medium- Term Maintenance	Long Term Maintenance
Module A	Pot A1	29.56	25.86
Module A	Pot A2	30.02	26.33
Module A	Pot A3	29.65	25.95
Module A	Pot A4	29.56	25.86
Module B	Pot B1	90.15	78.69
Module B	Pot B2	90.64	79.19
Module B	Pot B3	88.56	77.11
Module B	Pot B4	91.64	80.18
Module C	Pot C1	45.4	39.86
Module C	Pot C2	46.15	40.6
Module C	Pot C3	45.94	40.39
Module C	Pot C4	44.34	38.8
Module D	Pot D1	94.39	82.57
Module D	Pot D2	95.19	83.37
Module D	Pot D3	94.89	83.07
Module D	Pot D4	94.59	82.77

.

4. Conclusions

The overall equipment efficiency of the curing machine is articulated using the availability of all modules of the curing machine. The trained RL is implemented by calculation of availability for scheduled and unscheduled maintenance. In the degraded state, the time taken for unplanned maintenance is not predicted, since failure identification of components is unforeseeable in this machine. Effective maintenance is done in the form of intelligent assessment of durability, which is carried out on all modules of machines and reduces the probability of downtime. Automated maintenance is proposed using an information system for planned and unplanned maintenance. The information system contains maintenance history and maintenance forecasting, and reveals the improvement in product quality and quantity of the curing machine at the production site. The overall equipment efficiency obtained in Module D Pot D₂ Machine is 95.19% for short- and medium-term maintenance, and 83.37% for long-term maintenance, which is better than existing industrial methods. The RL technique used in the information system for short-, medium-, and long-term maintenance has better results than existing techniques. Improving the availability of the machine has made remarkable profits for the industry without considering any rejection due to machine maintenance. In future, the RL technique is recommended not only for the rubber industry, but also for all types of smart industry that demand Industry 4.0 technology.