A Novel Fault Detection with Minimizing the Noise-Signal Ratio Using Reinforcement Learning

Abstract

In this paper, a reinforcement learning approach is proposed to detect unexpected faults, where the noise-signal ratio of the data series is minimized to achieve robustness. Based on the information of fault free data series, fault detection is promptly implemented by comparing with the model forecast and real-time process. The fault severity degrees are also discussed by measuring the distance between the healthy parameters and faulty parameters. The effectiveness of the algorithm is demonstrated by an example of a DC-motor system.

1. Introduction

With the increasing expense and complexity of modern industrial systems, there is a growing demand for higher reliability and security. Measurement instrument faults may result in performance degradation or even malfunction due to the incorrect conclusion drawn by the process fault detection and diagnosis system. Therefore, the problem of fault detection and diagnosis (FDD) has become a popular research topic [1,2,3].

Generally, fault diagnosis methods can be categorized into model-based methods, signal-based methods and knowledge-based methods [1,2]. In model-based methods, the models of the industrial processes or the practical systems are obtained by using either physical principles or system identification techniques. Based on the model, fault diagnosis algorithms are developed to monitor the consistency between the measured outputs of the practical systems and the model-predicted outputs. Signal-based methods utilize measured signals rather than explicit input-output models for fault diagnosis. The feature signals to be extracted for symptom (or pattern) analysis can be either the time domain (e.g., mean, trends, standard deviation, phases, slope and magnitudes such as peak and root mean square) or frequency domain (e.g., spectrum). These issues were studied by various signal processing methods, such as wavelet transform (WT) [4], empirical mode decomposition (EMD) [5,6], intrinsic mode functions (IMF) [7] and local mean decomposition (LMD) [8]. A large volume of data has been more accessible with the development of modern electronic and measurement technologies such as SCADA and smart sensors [9,10,11,12,13], which stimulates knowledge-based fault diagnosis methods. Applying a variety of artificial intelligent techniques (either symbolic intelligence or computing intelligence) to the available historic data of the industrial processes, the underlying knowledge, which implicitly represents the dependence of the system variables, can be extracted. Interesting results on knowledge-based fault diagnosis and applications were reported during the last few decades [14,15,16,17,18].

Unexpected faults may cause performance degradation or even malfunction, and it is thus desired to detect, isolate and identify the faulty components as early as possible. However, it is difficult to release the fault feature in a short time because of the influences from heavy background noises. Based on the statistical theory, the traditional data-driven methods can be implemented by the sliding window technology in which the data are regarded as a concentration of system character and renew with window sliding. The features of the system can be extracted by analysing the data series in a sliding window after a filtering process and further stressed by strengthening technology such as PCA [19], SVM [20], information theory [21], and so forth. These traditional approaches have two flaws for fault detection: The first is that more data examples need to be collected in order to achieve a change of statistical character with a fault occurrence because a few new data can only have a small impact on the statistical character of the whole window. More data examples require more time to collect. Therefore, it is difficult for the traditional sliding window-based technology to carry out swift fault detection. The second is the lack of effective data in the case of early unexpected fault. Due to the complexity, uncertainty and unpredictability of the faults, it is challenging to obtain a number of valid fault data within a short period except for some special cases such as batch process. It is trade-off between getting more faulty data and giving less admissible time.

It is well known that the model parameters are more reliable than the state variables, especial in a noisy condition. However, the model parameters also face two problems similar to the aforementioned ones. The traditional approaches struggle to provide a quick detection due to the lack of the early information on sudden and unexpected faults.

Reinforcement learning (RL) is a powerful tool, which is motivated by statistics, psychology, neuroscience and computer science [22,23,24]. An agent will learn through experience, without a teacher. In each training session, named an episode, the agent explores the environment and receives the reward if any until it reaches the desired goal. The purpose of the training is to enhance the ‘brain’ of the agent. The goal of an agent is to maximize the reward that is received in the long run. One can obtain the optimal action only using the current states [25,26,27,28].

Motivated by the idea of “obtain the optimal action only using the current states”, an original idea based on RL is proposed to solve the swift fault detection problem. The minimization of the noise-signal ratio (NSR) is taken as the goal of the expecting series, and the policy iteration of RL is used as a tool to get parameters by considering the parameters as actions of RL. Then, one can get the model parameters corresponding to current states with noises. By comparing with the noise information (it is easier to get offline from the healthy data series), one will implement prompt fault detection and diagnosis with the next sample data. There are two main contributions in this paper.

(1) The unexpected faults will be detected promptly within a sampling period by using the measured data only.

(2) The estimated model is always consistent with the real-time process under the noisy condition by adjusting the parameters every sampling with the goal of minimizing the NSR using RL technology.

2. Problem Description and Preliminaries

2.1. Problem Description

Suppose a discrete-time system with noises is controlled by a pre-controller, depicted by Figure 1.

Here, $x(k-D),x(k-D+1),x(k),\cdots ,x(k+1)\in {\mathcal{R}}^n$ are the system states at sampling time $k-D,k-D+1,k,\cdots ,k+1$ respectively, and D is the order of the system. $u(k)\in {\mathcal{R}}^m,y(k)\in {\mathcal{R}}^p$ are the control input and measured output, respectively; $\omega (k)\in {\mathcal{R}}^n$ is a white Gaussian signal with zero mean and covariance matrix ${\sum }_{\omega }$. We suppose the system states are observable, and the control series $\{u(k)|u(k)\in {\mathcal{R}}^m,k=1,2,\cdots ,\}$ is obtained from the pre-controller’s output.

Let $\varphi (k)={[x^T(k-D)x^T(k-D+1)\cdots x^T(k)u^T(k)]}^T\in {\mathcal{R}}^{Dn+m}$; the system can be rewritten as a vector form:

$$x_m(k+1)={\theta }^T\varphi (k)+\omega (k)$$

(1)

where ${\theta }^T=\left[\begin{array}{ccc}{\theta }_1& \cdots & {\theta }_{1,Dn+m}\\ \cdots & \ddots & \cdots \\ {\theta }_{n,1}& \cdots & {\theta }_{n,Dn+m}\end{array}\right]\in {\mathcal{R}}^{n\times (Dn+m)}$ is a parameter matrix and T represents a transpose.

2.2. Noise-Signal Ratio

The noise is categorized into multiplicative noises and additive noise. Here, we only take into consideration additive noise, which is consistent with the nature of many processes. This means $x(k)=x^{\ast }(k)+\omega (k)$ for any time k, where $x(k)$ is the observed system states, $x^{\ast }(k)$ is the real data without noise and $\omega (k)$ is the noise.

Define a noise-signal ratio ${\delta }_i$ of i-th-component of data series $\{x(k)|x(k)\in {\mathcal{R}}^n,k=1,2,\cdots ,l\}$ as:

$${\delta }_i=\frac{\sqrt{{\sum }_{k=1}^l{[x_i(k)-x_i^{\ast }(k)]}^2}}{\sqrt{{\sum }_{k=1}^l{x}_i^{\ast }{(k)}^2}}$$

(2)

where $x_i(k)$ and $x_i^{\ast }(k)$ are the i-th component of the measured data and the real data at k sampling time, respectively, and l is the length of the data series. Further, an integer noise-signal ratio $\delta$ of data series $\{x(k)|x(k)\in {\mathcal{R}}^n,k=1,2,\cdots ,l\}$ for an additive noise is:

$$\delta =\sum _{i=1}^n{\delta }_i=\sum _{i=1}^n\frac{\sqrt{{\sum }_{k=1}^l{[x_i(k)-x_i^{\ast }(k)]}^2}}{\sqrt{{\sum }_{k=1}^l{x}_i^{\ast }{(k)}^2}}$$

(3)

There are three factors that affect the noise-signal ratio ${\delta }_i$ for a given n-dimensional data series: the measured data $\{x_i(k)\}$, the real data $x_i^{\ast }(k)$ and the length l. From the statistics viewpoint, l must have enough length in order to discover the feature of data series. This means it will spend a long time collecting the sample data. If one pursues a short time, the length l should be shorter. It is evident that when l becomes shorter, the noise will have a greater effect on the statistics character of the measured data series. It is a compromise between accuracy and velocity.

2.3. Reinforcement Learning Method

The reinforcement learning that is motivated by statistics, psychology, neuroscience and computer science is a powerful tool to deal with uncertain surroundings by interacting with its environment. In terms of [22,24,25], the basic theory and methods of the reinforcement-learning are simply introduced here. The basic frame of reinforcement learning is shown in Figure 2 [24].

An agent will get the evaluation of good or bad behaviour on the environment and learn through experience without a teacher, who teaches how to do perform this. In every single training session, named an episode, the agent explores the environment by changing action $u_i$ and receives the state $x_i$ and the reward $R_i$. The purpose of the training is to enhance the ‘brain’ of the agent. The goal of an agent is to maximize the reward $\sum R_i$ that is received in the long run.

Consider a Markov decision process $MDP(\mathcal{X},\mathcal{U},\mathcal{P},\Re )$, where $\mathcal{X}$ is a set of states and $\mathcal{U}$ is a set of actions or controls. The transition probabilities $\mathcal{P}$: $\mathcal{X}\times \mathcal{U}\times \mathcal{X}\to [0,1]$ represent for each state $x\in \mathcal{X}$ and action $u\in \mathcal{U}$ the conditional probability $P(x(k+1),x(k),u(k))=Pr\{x(k+1)\mid x(k),u(k)\}$ of transitioning to state $x(k+1)\in \mathcal{X}$ where the MDP is in state $x(k)$ and takes action $u(k)$. The cost function $\Re :\mathcal{X}\times \mathcal{U}\times \mathcal{X}\to \mathcal{R}$ is the expected immediate cost $R_k(x(k+1),x(k),u(k))$ paid after transition to state $x(k+1)\in \mathcal{X}$, given that the MDP starts from state $x(k)\in \mathcal{X}$ and takes action $u(k)\in \mathcal{U}$. The value of a policy $V_k^{\pi }(x(k))$ is defined as the conditional expected value of the future cost $E_{\pi }\{{\sum }_{i=k}^{k+T}{\gamma }^{i-k}{R}_i\}$, with $R_i\in \mathcal{R}$ when starting in state $x(k)$ at time k and following policy $\pi (x,u)$. One can further have:

$$\begin{array}{c}{V}_k^{\pi }(x)=E_{\pi }\{{\displaystyle \sum _{i=k}^{k+T}}{\gamma }^{i-k}{R}_i\}\hfill \\ ={\displaystyle \sum _u}\pi (x,u){\displaystyle \sum _{x(k+1)}}P(x(k+1),x(k),u(k))[R_k(x(k+1),x(k),u(k))+\gamma E_{\pi }\{{\displaystyle \sum _{i=k+1}^{k+T}}{\gamma }^{i-(k+1)}{R}_i\}]\hfill \\ ={\displaystyle \sum _u}\pi (x,u){\displaystyle \sum _{x(k+1)}}P(x(k+1),x(k),u(k))[R_k(x(k+1),x(k),u(k))+\gamma V_{k+1}^{\pi }(x(k+1))]\hfill \end{array}$$

(4)

where $T=\infty$. It is noted that $T=\infty$ represents that the Markov decision process has enough length to show its essential characteristic according to the statistical law. If it is too short, the $V_k^{\pi }(x)$ is prone to inaccuracy with few data. We usually use enough length l instead of ∞ in practical application.

Equation (4) releases the value function $V_k^{\pi }(x)$ for the policy $\pi (x,u)$ satisfying the Bellman Equation [29]:

$$V_k^{\pi }(x)=\sum _u\pi (x,u)\sum _{x(k+1)}P(x(k+1),x(k),u(k))[R_k(x(k+1),x(k),u(k))+\gamma V_{k+1}^{\pi }(x(k+1))]$$

(5)

Therefore, the optimal actions can be gained by alternating the policy evaluation and policy improvement according to Equations (6) and (7):

$$V_k(x)=\sum _u{\pi }_k(x,u)\sum _{x(k+1)}P(x(k+1),x(k),u(k))[R_k(x(k+1),x(k),u(k))+\gamma V_k(x(k+1))]$$

(6)

$${\pi }_k(x,u)=\underset{\pi }{argmin}\sum _{x(k+1)}P(x(k+1),x(k),u(k))[R_k(x(k+1),x(k),u(k))+\gamma V_k(x(k+1))]$$

(7)

where $\gamma$ is a discount factor with $0\le \gamma <1$ in order to be convergent.

For a deterministic system, ${\sum }_u{\pi }_k(x,u){\sum }_{x(k+1)}P(x(k+1),x(k),u(k))=1$. As a result, Equations (6) and (7) are rewritten as:

$$V_k(x)=R_k(x(k+1),x(k),u(k))+\gamma V_k(x(k+1))$$

(8)

$${\pi }_k(x,u)=\underset{\pi }{argmin}{R}_k(x(k+1),x(k),u(k))+\gamma V_k(x(k+1))$$

(9)

It is stressed that $x(k+1)$ is only a temporary expected state in the process of alternating the policy evaluation and policy improvement, which is used to implement the cost $R_k(x(k+1),x(k),u(k))$. The policy improvement (9) is usually obtained by using the greedy method [24] that will pursue the better policy at each iteration.

Remark 1.

There is only state information in Equations (8) and (9). One can obtain the optimal action only using the two states $x(k)$ and $x(k+1)$ in the process of minimizing the goal $R_k$. It does not need more time to collect more data, and the past information is not necessary to know.

3. Proposed Methodology

3.1. The System Reconfiguration and Parameter Acquisition

3.1.1. Fault-Free Scenario

One can obtain the estimated Equation of System (1) as follows:

$$\widehat{x}(k+1)={\widehat{\theta }}^T\varphi (k)={[{\widehat{\theta }}_1^T{\widehat{\theta }}_2^T\cdots {\widehat{\theta }}_n^T]}^T\varphi (k)$$

(10)

where $\widehat{x}(k+1)$ is an estimated value of $x(k+1)$; ${\widehat{\theta }}_1,\cdots ,{\widehat{\theta }}_n$ are vector components of $\widehat{\theta }$. If there are enough data in data series with length l, the parameter $\widehat{\theta }$ can be gained by using a least squares method (LSM) [30] according to the following:

$$\widehat{\theta }={[{\varphi }^T\varphi ]}^{-1}{\varphi }^T{x}_{k+1,l}$$

(11)

where $\varphi ={[{\varphi }_1,{\varphi }_2,\cdots ,{\varphi }_k,\cdots ,{\varphi }_l]}^T,{\varphi }_k={[x_{k-D,1},\cdots ,x_{k-D,n},\cdots ,x_{k,1},\cdots ,x_{k,n},u_{k,1},\cdots ,u_{k,m}]}^T\in {\mathcal{R}}^{Dn+m}$, $x_{k+1,l}={[x_{k+1,1},\cdots ,x_{k+1,l}]}^T$ and the subscripts k and $k+1$ are the sampling time instants, while l is the length of the data series. The accuracy of $\widehat{\theta }$ is further improved online by a recursion Equation (12) with new data $x_{k+1,l}$:

$$\begin{array}{c}{\widehat{\theta }}_{k+1}={\widehat{\theta }}_k+\frac{P_k\varphi [x_{k+1,l+1}-{\varphi }^T\widehat{\theta }]}{1+{\varphi }^T{P}_k\varphi }\hfill \\ P_{k+1}=P_k-\frac{P_k\varphi {\varphi }^T}{1+{\varphi }^T{P}_k\varphi }{P}_k\hfill \\ P_k=P_0\hfill \end{array}$$

(12)

where P is an auxiliary matrix and $P_0=\beta I$ for some large positive constant $\beta$; and ${\widehat{\theta }}_{k+1}$ is an estimated parameter improved by adding new data.

Goodwin and Sin [30] showed that LSM converges asymptotically to the true parameters if $\widehat{\theta }$ is fixed and $\varphi (k)$ satisfies the persistent excitation condition:

$${ϵ}_{0}I\le \frac{1}{N}\sum _{k=1}^N\varphi (k){\varphi }^T(k)\le {\overline{ϵ}}_{0}I$$

(13)

for all $N\ge N_0$, where ${ϵ}_0\le {\overline{\epsilon }}_0$ and $N_0$ is a positive number. This indicates $x^{\ast }=\widehat{x}$ in the meaning of the LSM. Here, $x^{\ast }$ is the real data without noise, and $\widehat{x}$ is an estimated value by using LSM.

3.1.2. Fault Scenario

It is assumed that the change from the normal to faulty operation does not affect the noise distribution and intensity. A model of data series subjected to a fault ${\omega }_f$ is described as:

$$x_f(k+1)={\theta }_f^T{\varphi }_f(k)+\omega (k)+{\omega }_f(k)$$

(14)

where ${\theta }_f\in {\mathcal{R}}^{Dn+m}$ is a coefficient vector after fault, $\omega (k)$ is the noise that is the same as fault free and ${\omega }_f$ is an unexpected fault. One can obtain ${\widehat{\theta }}_f$ by applying the least squares method again if there are enough valid data. The estimated model subjected to faults is as Equation (15):

$${\widehat{x}}_f(k+1)={\widehat{\theta }}_f^T{\varphi }_f(k)={[{\widehat{\theta }}_{f1}^T{\widehat{\theta }}_{f2}^T\cdots {\widehat{\theta }}_{fn}^T]}^T\varphi (k)$$

(15)

Substitute (10) and (15) into (2), hence the noise-signal ratio of fault free ${\delta }_i$ and of fault ${\delta }_{f,i}$ is Equations (16) and (17):

$${\delta }_i=\frac{\sqrt{{\sum }_{k=1}^l{[x_i(k)-{\widehat{\theta }}_i^T\varphi (k-1)]}^2}}{\sqrt{{\sum }_{k=1}^l{[{\widehat{\theta }}_i^T\varphi (k-1)]}^2}}$$

(16)

$${\delta }_{f,i}=\frac{\sqrt{{\sum }_{k=1}^l{[x_{fi}(k)-{\widehat{\theta }}_{fi}^T{\varphi }_f(k-1)]}^2}}{\sqrt{{\sum }_{k=1}^l{[{\widehat{\theta }}_{fi}^T{\varphi }_{fi}(k-1)]}^2}}$$

(17)

The integer noise-signal ratio of fault free $\delta$ and of fault ${\delta }_f$ is obtained by substituting (10) and (15) into (3):

$$\delta =\sum _{i=1}^n\frac{\sqrt{{\sum }_{k=1}^l{[x_i(k)-{\widehat{\theta }}_i^T\varphi (k-1)]}^2}}{\sqrt{{\sum }_{k=1}^l{[{\widehat{\theta }}_i^T\varphi (k-1)]}^2}}$$

(18)

$${\delta }_f=\sum _{i=1}^n\frac{\sqrt{{\sum }_{k=1}^l{[x_{fi}(k)-{\widehat{\theta }}_{fi}^T{\varphi }_f(k-1)]}^2}}{\sqrt{{\sum }_{k=1}^l{[{\widehat{\theta }}_{fi}^T{\varphi }_{fi}(k-1)]}^2}}$$

(19)

Remark 2.

The noise-signal ratio ${\delta }_{f,i}$ and ${\delta }_f$ subjected to fault has a similar form as the noise-signal ratio ${\delta }_i$ and δ that is fault free. One can get ${\widehat{\theta }}_i$ by the LSM method because there are enough valid data that are fault free. However, it is impracticable for ${\widehat{\theta }}_{fi}$ in the early fault due to lack of effective data subject to limited time.

The noise-signal ratio for a data series that is given a dimension n and a length l is related to three factors: the current measured data $\{x(k)\}$, parameter ${\widehat{\theta }}_i$ and the historical inputs $\varphi (k-1)$ in the condition of either fault or fault free. When $l=1$, Equation (19) becomes Equation (20):

$${\delta }_f(k)=\sum _{i=1}^n\frac{\sqrt{{[x_{fi}(k)-{\widehat{\theta }}_{fi}^T{\varphi }_f(k-1)]}^2}}{\sqrt{{[{\widehat{\theta }}_{fi}^T{\varphi }_{fi}(k-1)]}^2}}$$

(20)

The noise-signal ratio ${\delta }_f(k)$ of single sample $x_{fi}(k)$ is referred to by using the input ${\varphi }_f(k-1)$ and responding parameter ${\widehat{\theta }}_{fi}^T$ at sample k. The other way around, one can get ${\widehat{\theta }}_{fi}^T$ at sample k by using ${\delta }_f(k)$ in the case of knowing $x_{fi}(k)$ and ${\varphi }_f(k-1)$.

3.2. The Relation between Noise-Signal Ratio and Parameter

Theorem 1.

For a data series $\{x(k),k=0,\cdots ,l\}$, the following conclusions are obtained if it is written as the form of Equation (1):

1. Different ${\omega }_f$ induce different ${\widehat{\theta }}_f$;

2. The same ${\widehat{\theta }}_f$ causes the same noise-signal ratio ${\delta }_f$;

3. Different ${\widehat{\theta }}_f$ incurs different noise-signal ratio ${\delta }_{f,i}$.

Proof. 

1. For a measured data series $\{x(0),x(1),\cdots ,x(k),\cdots ,x(l)\}$ subjected to fault and noise, it can be described by:

$$x(k+1)={\widehat{\theta }}_f^T\varphi (k)+\omega (k)+{\omega }_f(k)$$

(21)

where ${\widehat{\theta }}_f$ is the parameter by LSM and $\varphi (k)={[x^T(k-D),x^T(k-D+1)\cdots x^T(k)]}^T$.

For a fault denoted by ${\omega }_{f1}(k)$, the data series can be written as:

$$x_{f1}(k)={\widehat{\theta }}_{f1}^T{\varphi }_{f1}(k-1)+\omega (k)+{\omega }_{f1}(k)$$

(22)

For a fault denoted by ${\omega }_{f2}(k)$, we are not sure whether the fault will change the parameter ${\widehat{\theta }}_f$. Therefore, the data series can be written as:

$$x_{f2}(k)={\widehat{\theta }}_{f2}^T{\varphi }_{f2}(k-1)+\omega (k)+{\omega }_{f2}(k)$$

(23)

where the subscripts $f_1$ and $f_2$ are used to distinguish the data and parameters under different faults.

It is noted that we discuss the data properties of a measured data series. As a result, $x_{f1}(k)=x_{f2}(k)$ and ${\varphi }_{f1}(k-1)={\varphi }_{f2}(k-1)$.

We assume ${\widehat{\theta }}_{f1}={\widehat{\theta }}_{f2}$ when ${\omega }_{f1}\ne {\omega }_{f2}$. Therefore, we can have:

$$0=x_{f1}(k)-x_{f2}(k)={\widehat{\theta }}_{f1}^T{\varphi }_{f1}(k-1)+\omega (k-1)+{\omega }_{f1}(k-1)-{\widehat{\theta }}_{f2}^T{\varphi }_{f2}(k-1)-\omega (k-1)-{\omega }_{f2}(k-1)$$

(24)

leading to ${\omega }_{f1}={\omega }_{f2}$, which is contradiction. As a result, we can have ${\widehat{\theta }}_{f1}\ne {\widehat{\theta }}_{f2}$ when ${\omega }_{f1}\ne {\omega }_{f2}$.

2. According to the definition of Equation (2), we have:

$$\frac{{\delta }_{f1,i}}{{\delta }_{f2,i}}=\frac{\sqrt{\frac{{\sum }_{k=1}^l{[x_{f1,i}(k)-{\widehat{\theta }}_{f1i}^T{\varphi }_{f1}(k-1)]}^2}{{\sum }_{k=1}^l{[{\widehat{\theta }}_{f1}^T{\varphi }_{f1}(k-1)]}^2}}}{\sqrt{\frac{{\sum }_{k=1}^l{[x_{f2,i}(k)-{\widehat{\theta }}_{f2i}^T{\varphi }_{f2}(k-1)]}^2}{{\sum }_{k=1}^l{[{\widehat{\theta }}_{f2i}^T{\varphi }_{f2}(k-1)]}^2}}}$$

(25)

For a measured data series $\{x(0),x(1),\cdots ,x(k),\cdots ,x(l)\}$, it is noted that $x_{f1}(k)=x_{f2}(k)$ and ${\varphi }_{f1}(k-1)={\varphi }_{f2}(k-1)$. For ${\widehat{\theta }}_{f1}={\widehat{\theta }}_{f2}$, one thus has:

$$\sqrt{\frac{{\sum }_{k=1}^l{[x_{f1,i}(k)-{\widehat{\theta }}_{f1i}^T{\varphi }_{f1}(k-1)]}^2}{{\sum }_{k=1}^l{[{\widehat{\theta }}_{f1}^T{\varphi }_{f1}(k-1)]}^2}}=\sqrt{\frac{{\sum }_{k=1}^l{[x_{f2,i}(k)-{\widehat{\theta }}_{f2i}^T{\varphi }_{f2}(k-1)]}^2}{{\sum }_{k=1}^l{[{\widehat{\theta }}_{f2i}^T{\varphi }_{f2}(k-1)]}^2)}}$$

(26)

It is obvious that ${\delta }_{f1,i}\ne 0$ and ${\delta }_{f2,i}\ne 0$. Therefore, $\frac{{\delta }_{f1}}{{\delta }_{f2}}=1$. Therefore, ${\delta }_{f1}={\delta }_{f2}$. This means the same ${\widehat{\theta }}_f$ causes the same noise-signal ratio ${\delta }_{f,i}$. Further, it results in the same integer noise-signal ratio ${\delta }_f$ due to ${\delta }_f={\sum }_{i=1}^n{\delta }_{f,i}$ according to Equation (3).

3. Arbitrary select the i-th component $x_{f,i}$ of $x_f$.

Hypothesis 1. 

Different ${\widehat{\theta }}_f$ have the same noise-signal ratio ${\delta }_f$, which means ${\delta }_{f1}={\delta }_{f2}$. Observe that:

$$\sqrt{\frac{{\sum }_{k=1}^l{[x_1(k)-{\widehat{\theta }}_{f1}^T\varphi (k-1)]}^2}{{\sum }_{k=1}^l{[{\widehat{\theta }}_{f1}^T\varphi (k-1)]}^2}}=\sqrt{\frac{{\sum }_{k=1}^l{[x_2(k)-{\widehat{\theta }}_{f2}^T\varphi (k-1)]}^2}{{\sum }_{k=1}^l{[{\widehat{\theta }}_{f2}^T\varphi (k-1)]}^2}}$$

(27)

which is equivalent to:

$$\frac{{\sum }_{k=1}^l{[x_1(k)-{\widehat{\theta }}_{f1}^T\varphi (k-1)]}^2}{{\sum }_{k=1}^l{[{\widehat{\theta }}_{f1}^T\varphi (k-1)]}^2}-\frac{{\sum }_{k=1}^l{[x_2(k)-{\widehat{\theta }}_{f2}^T\varphi (k-1)]}^2}{{\sum }_{k=1}^l{[{\widehat{\theta }}_{f2}^T\varphi (k-1)]}^2}=0$$

(28)

Rearranging the Equation above, we have:

$$\sum _{k=1}^l{[{\widehat{\theta }}_{f2}^T\varphi (k-1)]}^2\sum _{k=1}^l{[x_1(k)-{\widehat{\theta }}_{f1}^T\varphi (k-1)]}^2-\sum _{k=1}^l{[{\widehat{\theta }}_{f1}^T\varphi (k-1)]}^2\sum _{k=1}^l{[x_2(k)-{\widehat{\theta }}_{f2}^T\varphi (k-1)]}^2=0$$

(29)

$$\sum _{k=1}^l\sum _{j=1}^l{[{\widehat{\theta }}_{f2}^T\varphi (k-1)]}^2{[x_1(j)-{\widehat{\theta }}_{f1}^T\varphi (j-1)]}^2-\sum _{k=1}^l\sum _{j=1}^l{[{\widehat{\theta }}_{f1}^T\varphi (k-1)]}^2{[x_2(j)-{\widehat{\theta }}_{f2}^T\varphi (j-1)]}^2=0$$

(30)

$$\sum _{k=1}^l\sum _{j=1}^l\{{[{\widehat{\theta }}_{f2}^T\varphi (k-1)]}^2{[x_1(j)-{\widehat{\theta }}_{f1}^T\varphi (j-1)]}^2-{[{\widehat{\theta }}_{f1}^T\varphi (k-1)]}^2{[x_2(j)-{\widehat{\theta }}_{f2}^T\varphi (j-1)]}^2\}=0$$

(31)

Denote:

$$\begin{array}{c}\hslash ={[{\widehat{\theta }}_{f2}^T\varphi (k-1)]}^2{[x_1(j)-{\widehat{\theta }}_{f1}^T\varphi (j-1)]}^2-{[{\widehat{\theta }}_{f1}^T\varphi (k-1)]}^2{[x_2(j)-{\widehat{\theta }}_{f2}^T\varphi (j-1)]}^2\hfill \\ =\underset{\hslash 1}{\underbrace{{\{{\widehat{\theta }}_{f2}^T\varphi (k-1)[x_1(j)-{\widehat{\theta }}_{f1}^T\varphi (j-1)]-{\widehat{\theta }}_{f1}^T\varphi (k-1)[x_2(j)-{\widehat{\theta }}_{f2}^T\varphi (j-1)]\}}^2}}\hfill \\ +\underset{\hslash 2}{\underbrace{2{\widehat{\theta }}_{f2}^T\varphi (k-1)[x_1(j)-{\widehat{\theta }}_{f1}^T\varphi (j-1)]{\widehat{\theta }}_{f1}^T\varphi (k-1)[x_2(j)-{\widehat{\theta }}_{f2}^T\varphi (j-1)]-2{[{\widehat{\theta }}_{f1}^T\varphi (k-1)]}^2{[x_2(j)-{\widehat{\theta }}_{f2}^T\varphi (j-1)]}^2}}\hfill \end{array}$$

(32)

by the matching squares method.

Let:

$$\hslash 1={{\widehat{\theta }}_{f2}^T\varphi (k-1)[x_1(j)-{\widehat{\theta }}_{f1}^T\varphi (j-1)]-{\widehat{\theta }}_{f1}^T\varphi (k-1)[x_2(j)-{\widehat{\theta }}_{f2}^T\varphi (j-1)]}^2\ge 0$$

(33)

$$\begin{array}{c}\hfill \hslash 2=2{\widehat{\theta }}_{f2}^T\varphi (k-1)[x_1(j)-{\widehat{\theta }}_{f1}^T\varphi (j-1)]{\widehat{\theta }}_{f1}^T\varphi (k-1)[x_2(j)-{\widehat{\theta }}_{f2}^T\varphi (j-1)]-2{[{\widehat{\theta }}_{f1}^T\varphi (k-1)]}^2{[x_2(j)-{\widehat{\theta }}_{f2}^T\varphi (j-1)]}^2\\ \hfill =2\{\underset{\hslash 4}{\underbrace{{\widehat{\theta }}_{f2}^T\varphi (k-1)[x_1(j)-{\widehat{\theta }}_{f1}^T\varphi (j-1)]-{\widehat{\theta }}_{f1}^T\varphi (k-1)[x_2(j)-{\widehat{\theta }}_{f2}^T\varphi (j-1)]}}\}\underset{\hslash 3}{\underbrace{{\widehat{\theta }}_{f1}^T\varphi (k-1)[x_2(j)-{\widehat{\theta }}_{f2}^T\varphi (j-1)]}}\end{array}$$

(34)

Further, let:

$$\hslash 3=\underset{x_1(k)}{\underbrace{({\widehat{\theta }}_{f1}^T\varphi (k-1))}}\underset{\omega (j-1)}{\underbrace{[x_2(j)-{\widehat{\theta }}_{f2}^T\varphi (j-1)]}}$$

(35)

Note $\hslash 3\ne 0$ besides $x_1(k)=0$ or $\omega (j-1)=0$ (scarcely):

$$\hslash 4={\widehat{\theta }}_{f2}^T\varphi (k-1)[x_1(j)-{\widehat{\theta }}_{f1}^T\varphi (j-1)]-[{\widehat{\theta }}_{f1}^T\varphi (k-1)][x_2(j)-{\widehat{\theta }}_{f2}^T\varphi (j-1)]=[{\widehat{\theta }}_{f2}^T-{\widehat{\theta }}_{f1}^T]\varphi (k-1)x_1(j)$$

(36)

$(x_1(j)=x_2(j))$ for the same series data.

If $\hslash =0$, there is $\hslash 1=0$ and $\hslash 4=0$. Further:

$$[{\widehat{\theta }}_{f2}^T-{\widehat{\theta }}_{f1}^T]\varphi (k-1)=0$$

(37)

Notice ${\widehat{\theta }}_{f2}^T$ and ${\widehat{\theta }}_{f1}^T$ are fixed by LSM and $\varphi (k-1)$ is a vector from measured data, but it is uncertain for all k. There is no other vector to satisfy this Equation except ${\widehat{\theta }}_{f2}^T-{\widehat{\theta }}_{f1}^T=0$. Therefore, ${\widehat{\theta }}_{f1}={\widehat{\theta }}_{f2}$, which is contrary to the hypothesis.

☐

Remark 3.

The above analyses release the relationship between parameters ${\widehat{\theta }}_f$ and noise-signal ratio ${\delta }_{f,i}$. One can eliminate the influence of noise by the most extent by adjusting the parameter ${\widehat{\theta }}_f$ with the target of minimizing the NSR of data series. Once the parameter ${\widehat{\theta }}_f$ is determined, the model is used to forecast the next state ${\widehat{x}}_{k+1}$ without noise. Therefore, the measured state $x_{k+1}$ will be judged immediately according to the noise law based on the model prediction ${\widehat{x}}_{k+1}$.

The parameters ${\widehat{\theta }}_f$ can be estimated by traditional methods such as LSM and MLE (maximum likelihood method) based on the historical numerical data. Window technology is used to reduce computational load, and the sliding window is employed to capture the time-varying parameters in the dynamic system. The statistics characteristics depend on the data in the window. A longer window, which includes more data, means higher accuracy, but needs more time to make a decision. A shorter window, which consists of less data, means a quick decision, but it also needs enough data in order to satisfy the statistics law.

3.3. Seeking ${\widehat{\theta }}_f$ by the Reinforcement Learning Method

Engineering systems are subjected to faults or malfunctions due to unexpected events, which would degrade the operation performance and even lead to the operation failure. As a result, the fault should be detected quickly, and measures will be taken as early as possible. The greatest difficulty is the lack of enough valid data for an early fault. Reinforcement learning provides a way to estimate the parameters directly by approaching the noise-signal ratio ${\delta }_f$ of the fault to noise-signal ratio ${\delta }_h$ of health (fault free).

To apply the reinforcement learning, the first thing is to determine the cost function $R_k({\delta }_f(k))$ at time k. Here, one defines the cost function $R_k({\delta }_f(k))$ at time k as an absolute value of error between the current integer noise-signal ratio ${\delta }_f(k)$ and the integer noise-signal ratio ${\delta }_h$ of being fault free.

$$R_k({\delta }_f(k))=\mid {\delta }_f(k)-{\delta }_h\mid =\sum _{i=1}^n\frac{\sqrt{{[x_{fi}(k)-{\widehat{\theta }}_{fi}^T{\varphi }_f(k-1)]}^2}}{\sqrt{{[{\widehat{\theta }}_{fi}^T{\varphi }_{fi}(k-1)]}^2}}-{\delta }_h$$

(38)

where ${\delta }_h$ is the integer noise-signal ratio of being fault free that will be achieved offline according to Equation (18), $|·|$ is the absolute value and the meanings of other parameters are the same as before. The function $V_k({\delta }_f(k))$ after time k is defined as:

$$V_k({\delta }_f(k))=\sum _{i=k}^{\infty }{\gamma }^{i-k}{R}_i({\delta }_f(i))$$

(39)

As a result, one has:

$$V_k({\delta }_f(k))=R_k({\delta }_f(k))+\gamma V_{k+1}({\delta }_f(k+1))$$

(40)

Following a Bellman optimal principle, the optimal value function is obtained according to Equation (41):

$$V^{\ast }({\delta }_f(k))=\underset{{\widehat{\theta }}_f(k)}{min}{R}_k({\delta }_f(k))+\gamma V_{k+1}({\delta }_f(k+1))$$

(41)

where $V^{\ast }({\delta }_f(k))$ and ${\widehat{\theta }}_f(k)$ are the optimal value function and the parameter at time k, respectively; and γ is a discount factor, $0\le \gamma <1$.

It is noticed that (41) cannot be used online because one cannot know the information of the future time instant, that is ${\delta }_f(k+1)$. A Q-algorithm proposed by Watkins [23] provides an effective solution by substituting the Q-function. A mimic of the Q-algorithm defines the evaluation function $Q({\delta }_f(k),{\widehat{\theta }}_f(k))$ as the minimum discounted cumulative reward that can be achieved from ${\delta }_f(k)$ and ${\widehat{\theta }}_f(k)$ as the first action:

$$Q({\delta }_f(k),{\widehat{\theta }}_f(k))\stackrel{def}{=}{R}_k({\delta }_f(k),{\widehat{\theta }}_f(k))+V^{\ast }(\phi ({\delta }_f(k),{\widehat{\theta }}_f(k)))$$

(42)

where $\phi ({\delta }_f(k),{\widehat{\theta }}_f(k))$ expresses the state ${\delta }_f(k+1)$ that comes from ${\delta }_f(k)$ and ${\widehat{\theta }}_f(k)$, that is ${\delta }_f(k+1)=\phi ({\delta }_f(k)$, ${\widehat{\theta }}_f(k))$. One denotes $\phi ({\delta }_f(k),{\widehat{\theta }}_f(k))$ in order to stress the relation between ${\delta }_f(k+1)$ and ${\delta }_f(k),{\widehat{\theta }}_f(k)$. If Q achieves its optimization under some parameter ${\widehat{\theta }}_f(k)$, the function V can also achieve its optimization with the same parameter. As a result, V may be replaced by Q. This implies that the optimal parameter can be obtained only by reward without using the value function V.

Denote the optimum of Q as $Q^{\ast }$; therefore, one has:

$$\begin{array}{c}{Q}^{\ast }({\delta }_f(k),{\widehat{\theta }}_f(k))=\underset{{\widehat{\theta }}_f(k)}{min}[R_k({\delta }_f(k),{\widehat{\theta }}_f(k))+V^{\ast }(\phi ({\delta }_f(k),{\widehat{\theta }}_f(k)))]\hfill \\ =R^{\ast }({\delta }_f(k),{\widehat{\theta }}_f(k))+V^{\ast }({\delta }_f(k+1))=V^{\ast }({\delta }_f(k),{\widehat{\theta }}_f(k))\hfill \end{array}$$

(43)

where the superscript * expresses the optimal values. It is seen from Equation (43) that $Q^{\ast }({\delta }_f(k),{\widehat{\theta }}_f(k))$ is equivalent to $V^{\ast }({\delta }_f(k),{\widehat{\theta }}_f(k))$ with the same parameter. Therefore, the optimal parameter ${\widehat{\theta }}_f(k)$ can be obtained by the policy iteration that includes the alternation of two processes: policy evaluation and policy improvement following Equations (44) and (45):

$$Q({\delta }_f(k),{\widehat{\theta }}_f(k))=R({\delta }_f(k),{\widehat{\theta }}_f(k))+\gamma \underset{{\widehat{\theta }}_f(k+1)}{min}Q({\delta }_f(k+1),{\widehat{\theta }}_f(k+1))$$

(44)

$${\pi }_k({\delta }_f(k),{\widehat{\theta }}_f(k))=\underset{{\widehat{\theta }}_f(k+1)}{argmin}Q({\delta }_f(k),{\widehat{\theta }}_f(k))$$

(45)

where ${\pi }_k$ is called a policy in reinforcement learning. By using policy iteration, it will finally converge to the steady state, and we get the responding parameter.

It is important for policy iteration to be convergent. Fortunately, it has been proven by Lemma 1.

Lemma 1 ([21]).

Consider a Q learning agent in a deterministic Markov decision process (MDP) with bounded reward $(\forall {\delta }_f(k),{\widehat{\theta }}_f(k))\left|R_k({\delta }_f(k),{\widehat{\theta }}_f(k))\right|\le c$. The Q learning agent uses the training rule of Equation:

$$Q_k({\delta }_f(k),{\widehat{\theta }}_f(k))\leftarrow R_k({\delta }_f(k),{\widehat{\theta }}_f(k))+\gamma \underset{{\widehat{\theta }}_f(k+1)}{min}{Q}_{k+1}({\delta }_f(k+1),{\widehat{\theta }}_f(k+1))$$

initializes its $Q_k({\delta }_f(k),{\widehat{\theta }}_f(k))$ to arbitrary finite values and uses a discount factor γ such that $0\le \gamma <1$. Let $Q_k^{(n)}({\delta }_f(k),{\widehat{\theta }}_f(k))$ denote the agent’s hypothesis $Q_k({\delta }_f(k),{\widehat{\theta }}_f(k))$ following the n-th update. If each state-action pair is visited infinitely often, then $Q_k^{(n)}({\delta }_f(k),{\widehat{\theta }}_f(k))$ converges to $Q_k({\delta }_f(k),{\widehat{\theta }}_f(k))$ as $n\to \infty$, for all ${\delta }_f(k),{\widehat{\theta }}_f(k)$.

Remark 4.

Lemma 1 provides a guarantee on the convergence of Q learning. By using policy iteration, the Q learning agent will finally converge to the steady state, and the optimal control ${\pi }^{\ast }({\delta }_f(k),{\widehat{\theta }}_f(k))$ can be obtained readily.

Procedure 1:

The RL algorithm can be summarized as follows:

Step 1: Initialize $\widehat{Q}({\delta }_f(k),{\widehat{\theta }}_f(k))$ to zero.

Step 2: Select a parameter ${\widehat{\theta }}_f(k)$ randomly.

Step 3: Receive immediate reward $R({\delta }_f(k),{\widehat{\theta }}_f(k))$ according to Equation (38).

Step 4: Get the new state ${\delta }_f(k+1)$ and compute the value function according to Equation (40).

Step 5: Update the $\widehat{Q}({\delta }_f(k+1),{\widehat{\theta }}_f(k+1))$ based on current state ${\delta }_f(k)$ according to Equation (41).

Step 6: Set the next state ${\delta }_f(k+1)$ as the current state ${\delta }_f(k)$.

Step 7: Repeat Steps 3–6 until it is convergent.

Step 8: Find the best parameter ${\widehat{\theta }}_f^{\ast }(k)$ according to Equation (46).

3.4. Detection of Fault

Based on the parameters ${\widehat{\theta }}_f^{\ast }(k)$, we will get the next state ${\widehat{x}}_f(k+1)$ according to Equation (15). Therefore, we have a chance to judge new measure data $x_f(k+1)$ immediately with taking ${\widehat{x}}_f(k+1)$ as a criterion.

The state ${\widehat{x}}_f(k+1)$ with fault is made up of three parts: the real state $x^{\ast }(k+1)$ that is fault free, the component from fault ${\omega }_f$ and the component from noise ω. We take the first two items as an integer and remark that they are the real data $x_f^{\ast }(k+1)$ of $x_f(k+1)$. Considering the parameter ${\widehat{\theta }}_f^{\ast }$ is obtained by seeking for a goal of minimizing the noise-signal ratio, Equation (15) implies the noise minimization of forecasting the state at the next time $k+1$. Therefore, $x_f^{\ast }(k+1)$ is obtained by ${\widehat{\theta }}_f^{\ast }$ according to Equation (15). We will get the estimated state ${\widehat{x}}_f(k+1)$ at time $k+1$ in the case of fault according to:

$${\widehat{x}}_f(k+1)={\widehat{x}}_f^{\ast }(k+1)+e={({\widehat{\theta }}_f^{\ast })}^T(k){\varphi }_f(k)+e$$

(46)

where ${\widehat{\theta }}_f^{\ast }(k)$ is the parameters at time k obtained from the RL algorithm, T is the transpose and $e={[e_1,e_2,\cdots ,e_n]}^T$ is the confidence interval of noise ω at confidence level α:

$$e_i=\pm \sqrt{\frac{D_i}{l}}{Z}_{\alpha /2}$$

(47)

where $D_i$ is the variance of the i-th component of samples, which are obtained offline by data series $\{x(k)|x(k)\in R^n,k=1,2,\cdots ,l\}$ that is fault free:

$$D_i=\frac{1}{l-1}\sum _{j=1}^l{[x_i(j)-{\widehat{x}}_i(j)]}^2$$

(48)

$Z_{\alpha /2}$ is a normal distribution.

The above analysis shows that one can forecast ${\widehat{x}}_f(k+1)$ in a noisy condition only by using $\varphi (k)$ during one sampling period. It is valuable for the system to detect faults promptly.

Define the Euclidean distance ($ED$) between measure $x_f(k+1)$ and estimation ${\widehat{x}}_f(k+1)$ as:

$$ED(k+1)=\sum _{i=1}^n{[x_{fi}(k+1)-{\widehat{x}}_{fi}(k+1)]}^2$$

(49)

where $x_f(k+1)\in {\mathcal{R}}^n$ and ${\widehat{x}}_f(k+1)\in {\mathcal{R}}^n$ are the measured data and the estimated data at time $k+1$ under the fault, respectively.

The threshold of $ED$ is selected as the maximum error between measured data and estimated data being fault free:

$$E{D}_{sh}=\underset{k}{max}\{\sum _{i=1}^n{[x_i(k)-{\widehat{x}}_i(k)]}^2,k=1,\cdots ,l\}$$

(50)

One can detect a fault if:

$$ED(k+1)>E{D}_{sh}$$

(51)

Once one detects a fault, the parameters that are fault free will keep unchanged in order to build a virtual healthy model. Meanwhile, the parameters subject to fault continue to renew by the proposed RL method and forecast the next state under fault. In this condition, the $ED$ becomes an indicator of fault degree $(IFD)$. Therefore, we get Equation (52) by replacing ${\widehat{x}}_f(k+1)$ for fault with $\widehat{x}(k+1)$ for fault free in Equation (49):

$$IFD(k+1)=\sum _{i=1}^n{[{\widehat{x}}_{fi}(k+1)-\widehat{x}(k+1)]}^2$$

(52)

Here, ${\widehat{x}}_{fi}(k+1)$ for minimizing NSR is used to instead of $x_{fi}(k+1)$ in order to reduce the effect of noise. We use the $IFD(k+1)$ to express the severity of the fault at $k+1$, so we will evaluate the fault degree in time and take measures to balance the safety and efficiency of the plant.

Remark 5.

One will detect a fault and evaluate the fault degree promptly during one sampling period according to Equations (51) and (52).

The forecast of states at $k+1$ is valid under faulty or under fault-free conditions because the parameters of a reference model are essentially obtained by minimizing the noise-signal ratio.

This method only makes use of the residual and noise-signal ratio so that it is easy to identify the condition under being fault free. Meanwhile, it has the ability to trace unexpected fault by adjusting the parameters online.

Procedure 2:

The fault detection and fault seriousness degree procedure is given as follows:

Step 1. Get the next real state ${\widehat{x}}_f^{\ast }(k+1)$ without the noise based on the parameters ${\widehat{\theta }}_f^{\ast }(k)$ from Procedure 1 according to Equation (15).

Step 2. Computer the variance of the i-th component of samples from the data series being fault free according to Equation (48).

Step 3. Get the estimated state ${\widehat{x}}_f(k+1)$ according to Equation (7).

Sept 4. Get the measured data $x_f(k+1)$.

Step 5. Compute the Euclidean distance (ED) between measure $x_f(k+1)$ and estimation ${\widehat{x}}_f(k+1)$ according to Equation (49).

Step 6. Compute the threshold of ED according to Equation (50).

Step 7. Perform fault detection and get the fault seriousness degree according to (51) and (52).

Step 8. Go to Step 1 to check the next state.

4. Examples and Simulations

In this section, simulation results based on a DC-motor are presented to verify the efficacy of the proposed scheme. Figure 3 shows the topology of the DC-motor test bed.

The DC-motor is selected as Model 57BL90-210 with 24 V, 1000 rpm and 60 W. The rotary encoder is LPD3806-600BM. The integrated driver is an improved ZD-6405 that provides the positive inversion with a toggle switch and speed governing with 0–5V control voltage. It also gives the armature current detection and some protections against short circuit, under voltage and overload. The DC-motor is driven by an integrated driver with the controller of the STM32 single-chip microcomputer. The controller of STM32 is used to receive the DC-motor speed collected by the rotary encoder and the armature current from the integrated driver and, meanwhile, to output the driver control voltage according to the control approach. The controller is programmed on the plat of Keil3.0 by the JTAG (Joint-Test-Action-Group) interface, and the data are transmitted to the computer online in order to save memory. The computer is an i5-2320 CPU with 3.0 GHz and 32 G RAM. The MATLAB 2011 is used to run the method and share the data from the controller by data/file exchange technology. We add a white noise to data from the sensor before they are transmitted to the computer in order to strengthen the noise’s effects. The test bed of the DC-motor is shown in Figure 4.

A fault-free time series is produced according to the DC-motor system. The estimated model of one-order of system is obtained by LSM and has passed the statistical test under the significance level of 0.05 in the healthy condition:

$$\left[\begin{array}{c}{i}_a(k+1)\\ {\omega }_{\beta }(k+1)\end{array}\right]=\left[\begin{array}{ccc}0.4261& 0.0030& 0.0123\\ 1.3430& 0.9910& 0.0329\end{array}\right]\left[\begin{array}{c}{i}_a(k)\\ {\omega }_{\beta }(k)\\ u(k)\end{array}\right]$$

(53)

4.1. Swift Detection

Firstly, we do an experiment to test the speediness of fault judgement. The fault signal ${\omega }_f$ with a step of amplitude 0.2 is added to State x2 from Sample 200. The results from Sample 195 to Sample 235 are shown in Figure 5. The blue curve, the red curve and the green curve are the data that are fault free, measured data subject to fault and estimated data by the proposed method, respectively. When the fault occurs, the system responds to the fault after two sampling periods due to the inertia. State x1 conforms to the healthy state (blue curve) due to the little influence of this fault. State x2 begins to deviate from the blue curve from Sample 203 and raises to 0.5 after seven sampling periods. The new stability that has a stable bias with the healthy state (blue curve) achieves at the time of system response the stability of the fault. The estimated data (green curve) for the RL method are obtained by immediately adjusting the model parameters along with minimizing the NSR. One can see that the green curve coincides with the red curve whether before and after a fault occurs.

In order to compare with the sliding window method (SLW), we determine an estimated $\widehat{\theta }$ instead of θ by LSM with the width of sliding window l = 50. The result is shown in Figure 5 as the black curve. The black curve shows that State x2 has a similar tendency as the green curve except with a delay. During the healthy stage, both SLW and RL methods have good performance in tracing measure data (red curve), and the SLW has less fluctuations than RL. When a fault appears, the SLM will experience a transient process similar to the green curve, raising from 0.3–0.5 after about 25 sampling periods, but not immediately. This means the SLM will have a longer delay to respond to the fault. The SLW method is good for the healthy process that has a stable statistical indicator. When there is a fault occurrence, the statistical indicators of the data series move to a new stable state to fit the fault after they suffer a transition change. This process depends on the fault style and intensity. Therefore, the SLW method cannot avoid the delay due to its necessary data collection to change the statistical indicators in the range of its window length. It can speed the judgement by shortening the window length. However, if the window length is too small, the statistical indicators will become unstable because the data of the window cannot express the feature of the data series. Our proposed RL method will make up for this condition.

We also show a training process of minimizing the noise-signal ratio by reinforcement learning. It is seen in Figure 6. The horizontal coordinate and vertical coordinate represent the episodes and the responding NSR, respectively. The discount factor γ is 0.95. Beginning with a parameter ${\widehat{\theta }}_f(k)$ randomly (as Procedure 1), the NSR will converge after a training of 8300 episodes, and one will get the required parameter ${\widehat{\theta }}_f(k)$ when it is convergent.

4.2. Fault Detection

A comprehensive fault signal ${\omega }_f$ combined with a step, a sine and a slope is added to State x2 in order to verify the fault diagnosis and detection ability of the proposed RL method. The fault signal is generated according to Equation (54):

$${\omega }_f(k)=\left\{\begin{array}{ll}0\hfill & 0<k\le 200\\ 0.2\hfill & 200<k\le 300\\ 0.2+0.15sin(\pi (k-300)/30)\hfill & 300<k\le 600\\ -0.001k+0.8& 600<k\le 800\\ 0\hfill & 800<k\le 1000\end{array}\right.$$

(54)

and shown in Figure 7.

The state ${\widehat{x}}_f(k+1)$ at time $k+1$ is estimated based on the observation ${\varphi }_f(k)$ at time k according to Equation (15) and in which ${\widehat{\theta }}_f$ is obtained by the proposed RL approach. The evolution of states from $k=100$ to $k=1000$ is shown in Figure 8. The blue curve, the red curve and the green curve are the data that are fault free, the measured data and the estimated data, respectively.

It is seen from Figure 8 that the estimated data (green curve) coincide with the measure data (red curve) throughout the process of different faults. In fact, the green curve is an estimation based on the measured data at the previous moment by using the proposed RL approach. It is produced a sampling period earlier than the red curve. We also compute the errors between measurement and estimation according to Equation (49) in order to show the accuracy. The mean of x1 and x2 between measured data and estimated data are 0.05 and 0.02, respectively, and the maximum error is 0.25 and 0.15. The result is seen in Figure 9.

If the data that are fault free are taken as a reference and the fault degree is expressed with the $IFD$ according to Equation (52), the threshold of $ED$ is obtained in the condition of being fault free based on the healthy data from 1–200 by Equation (50) and $E{D}_{sh}=0.0286$. Then, we compute the $IFD$s at every sampling time according to Equation (52). The results are shown in Figure 10. The blue curve and the red curve are the indicator of fault degree ($IFD$) and the threshold of $ED$, respectively.

Figure 10 shows the $IFD$ that is fault free is below the threshold. During the fault process, the $IFD$ that fluctuates with a limited range is above the threshold except some samples that are close to healthy data.

We will also know the fault severity at every sample by observing the $IF{D}^{\prime }$s scale. For example, the fault from Sample 200–Sample 300 is limited between 0.05 and 0.15, which means the fault is comparatively stable. At Samples 320, 380, 440 and 510, a peak appears respectively with a heavy fault over 0.3.

4.3. Influence of Disturbance

We give a step disturbance to State x2 by raising the control voltage at Sample 20. The evolution of states is shown in Figure 11. The blue curve, the red curve and the green curve are data without disturbance, measured data and estimated data by the proposed method, respectively. It is seen that the armature current almost keeps the initial state because there is no load change. The angular velocity (red curve) rises to 0.4 rad in response to this disturbance after a short transition. The proposed method gives an ample estimation (green curve) because the data with disturbance have enlarged the NSR more that without disturbance in a long enough process. From an inverse view, an ample estimation will be taken to make up the NSR without disturbance according to the proposed method. This shows the RL’s robustness in disturbance.

The proposed method cannot distinguish between faults and disturbances because it makes a decision only according to the NSR. In fact, the disturbance is eliminated by the closed loop of the control system. If the disturbance cannot be removed by the control system due to the fault, it is necessary for this disturbance to be handled as a special fault in order to keep the plant safe and effective.

5. Conclusions

Comparing a single sample datum with healthy data is the fastest way for fault detection. However, it can hardly be achieved because the noise of sample data will disturb the normal data. No one knows whether the discrepancy between sample data and healthy data comes from fault or comes from noise only according to a single collected datum. The statistical method needs a quantity of valid data; however, it is difficult to obtain them in the early stage of unexpected fault, which leads to a dilemma of prompt FDD. In order to solve these shortages, a reinforcement learning method has been proposed to estimate the model parameter by taking the parameter as a special action. Taking a minimization of the NSR as a goal of the data series, the model parameter can be obtained by applying the technology of the policy valuation and policy improvement. This method has the ability of getting rid of the noise’s influence and keeping consistency with the current situation. Furthermore, the FDD has been implemented by evaluating the residual of the real-time process data and pre-obtained healthy time-series data. The fault can be promptly detected with the help of the threshold from the healthy data series by only using the information within one sampling period.

In the future, further work will distinguish the slight fault signal from healthy data as quickly as possible and apply this method to an engineering-oriented real-time process.