A KPI-Based Probabilistic Soft Sensor Development Approach that Maximizes the Coefficient of Determination

Abstract

Advanced technology for process monitoring and fault diagnosis is widely used in complex industrial processes. An important issue that needs to be considered is the ability to monitor key performance indicators (KPIs), which often cannot be measured sufficiently quickly or accurately. This paper proposes a data-driven approach based on maximizing the coefficient of determination for probabilistic soft sensor development when data are missing. Firstly, the problem of missing data in the training sample set is solved using the expectation maximization (EM) algorithm. Then, by maximizing the coefficient of determination, a probability model between secondary variables and the KPIs is developed. Finally, a Gaussian mixture model (GMM) is used to estimate the joint probability distribution in the probabilistic soft sensor model, whose parameters are estimated using the EM algorithm. An experimental case study on the alumina concentration in the aluminum electrolysis industry is investigated to demonstrate the advantages and the performance of the proposed approach.

1. Introduction

With the increasing demands placed on industry, requiring a decrease in the defective rate of products, better economic efficiency, and improved safety, there has been a growing demand to develop and implement approaches that can improve the overall control strategy [1]. The first issue that needs to be solved is achieving accurate and real-time estimation of key performance indicators (KPIs) [2]. The difficulty is that these KPIs are usually not easy to measure, or the measurement has significant time delay. Even if some KPIs are measurable, due to the complexity and nonlinearity of modern industrial systems and their complex working conditions, the KPIs may be extremely unreliable [3]. One way to solve the above problems is to develop a soft sensor, which seeks to select a group of easier-to-measure secondary variables that are correlated with the required primary variables (i.e., KPIs in this paper), so that the system is capable of providing process information as often as necessary for control [4,5]. In the development of a successful soft sensor, a good process model is required. The process models can be divided into two major categories: first principles models and data-driven models [6,7]. Although it is desirable to apply mass and energy balances to build a complete first principles model, lack of process knowledge, plant–model mismatch, and nonlinear characteristics limit the applicability of such an approach to the simplest processes. As an alternative, data-driven soft sensors are developed from historical data without necessarily considering any outside process knowledge. Data-driven soft sensors, which solely use available process data to develop a model of the process, have recently attracted considerable attention and have been successfully applied in many fields [8], such as fault detection (FD) and process monitoring, that are important for many industrial processes. Serdio [9] introduced an improved fault detection approach based on residual signals extracted online from system models identified by high-dimensional measurements provided by the multisensor network. The data-driven system identification model can also be combined using multivariate orthogonal space transformations and vectorized time-series models to achieve enhanced residual-based fault detection in condition monitoring systems equipped with a multisensor network [10]. Shardt [11] proposed a data-driven design of a diagnostic-observer-based process monitoring method, which was extended to include the ability to detect changes given infrequent KPI measurements. Yan [12] and Gabrys [13] introduced the most popular data-driven soft sensor modelling techniques, as well as discussing some issues in soft sensor development and maintenance and their possible solutions. Data-driven methods can be divided into three categories: models based on statistical analysis, models based on statistical learning theory [14], and models based on artificial intelligence [15].

Of interest for this paper are models developed using statistical methods to extract the relevant information from the large amounts of industrial data that are produced by the complex processes. Statistical methods have been developed that can handle such large datasets and develop useful models. Common methods include principal component analysis (PCA) [16] and partial least squares (PLS) [17]. PCA is a powerful tool for data compression and information extraction that can simplify the model structure and improve the speed of operations. However, PCA can only deal with the correlations between vectors in the same matrix. To overcome this limitation, PLS was developed as an approach that models the correlation between independent variables and dependent variables. Since PLS only applies to linear systems or weakly nonlinear systems, many nonlinear PLS algorithms have been developed to handle nonlinear systems. The neural-network-based PLS algorithm [18] uses the nonlinear processing capability of a neural network to describe the relationship between variables. However, the determination of the network structure and the selection of network training algorithms are difficult problems. In addition, if there are too many datapoints, the model structure will be very complex and the accuracy will be difficult to guarantee.

On the other hand, considering that data-driven modeling methods use historical data for training, this raises the question of how to handle missing data. Along with issues such as the reliability of sensors and multirate sampling, missing data is common in practical industry process [19,20]. For example, in the aluminum electrolysis process, the alumina concentration is usually obtained manually by laboratory staff. Considering human factors and chemical examination equipment reliability, data loss occurs from time to time. In this case, this type of measurement has different effects on the soft sensor modeling process and state estimation performance. Therefore, in order to make the soft sensor more suitable for practical, complex industrial processes, the missing data problem needs to be taken seriously. Compared with the direct deletion of missing data, the data interpolation method [21] is better able to restore the real situation. Currently, data interpolation methods include the mean substitution method, the regression interpolation method, and the expectation maximization (EM) algorithm. Of these, the mean substitution method can cause biased estimates, and the regression interpolation method is built based on a complete data set, where the linear relationship between the variables with missing values and other variables is necessary, which, in many cases, cannot be satisfied. In fact, the EM algorithm has good practical value as an iterative algorithm for simplifying the maximum likelihood estimation when dealing with missing data in sample sets [22].

Recently, in order to evaluate the accuracy of the model output, the coefficient of determination approach has been considered. The coefficient of determination is the measurement of how well the regression model fits the data [23]. Feng [24] introduced the coefficient of determination as a criterion for comparing the best-wavelength partial least squares regression (PLSR) model with the full-wavelength model. Boyaci [25] used the coefficient of determination to evaluate the adulteration rate of coffee beans, thus ensuring coffee quality. However, these applications only consider the coefficient of determination as an evaluation index without applying it for the modeling process. In general, the coefficient of determination is a criterion that can evaluate the quality of a model and has a concise structure, so it is appropriate to apply it to the soft sensor development process to establish a simpler and more accurate model for complex industrial process.

Therefore, this paper develops a KPI-based soft sensor model with simple structure and high accuracy, using the coefficient of determination method, which also solves the missing data issue using the EM algorithm.

2. Background

2.1. The Gaussian Mixture Model

As a flexible and efficient tool for probabilistic data models, a Gaussian mixture model (GMM) can be used to define any complex probability distribution function and is, therefore widely used in many statistical data modelling applications. In this paper, GMM is used to approximate the joint probability distribution in the soft sensor probability model. The reason for introducing GMM is that, theoretically, any probability distribution can be approximated using the joint weighted Gaussian distribution [26].

If x represents a multidimensional random variable, then the joint probability distribution of the GMM is expressed as

$$p\left(x|\mathsf{\Theta }\right)={\displaystyle \sum _{l=1}^M{\alpha }_l}{p}_l\left(x|{\theta }_l\right)$$

(1)

where α_l is the mixing coefficient, which represents the prior probability of each mixed component; M is the number of mixed components; and ${\displaystyle \sum _{l=1}^M{\alpha }_l}=1$. $\mathsf{\Theta }=\left({\theta }_1,{\theta }_2,\cdots ,{\theta }_M\right)$ is the parameter vector of each mixed component, and each Gaussian probability density function p_l(x) is determined by the parameter θ_l = (μ_l, Σ_l), where μ_l is the mean and Σ_l is the covariance matrix. The GMM parameters α_l, μ_l, and Σ_l (l = 1, 2, …, M) are estimated using the EM algorithm.

2.2. The Expectation Maximization Algorithm

The EM algorithm is a maximum likelihood estimation method for solving model distribution parameters from “incomplete data” and was first introduced in [27]. Each iteration of the algorithm involves two steps, called the expectation step (E-step) and the maximization step (M-step).

2.2.1. E-Step

Given the observation data set X and the current parameters Γ⁽ⁱ⁾, the expectation of the log-likelihood function is called the Q-function which can be written as

$$Q(\mathsf{\Gamma },{\mathsf{\Gamma }}^{\left(i\right)})=E\left[\log p\left(X,|\mathsf{\Gamma }\right)|X,{\mathsf{\Gamma }}^{\left(i\right)}\right]$$

(2)

where γ can represent missing data due to observational conditions and other reasons, and can also refer to hidden variables. Since the direct optimization of the likelihood function is usually very difficult, the relationship between X, Γ, and γ can be established by introducing an additional variable γ to achieve the purpose of simplifying the likelihood function.

2.2.2. M-Step

A new parameter Γ⁽ⁱ⁺¹⁾ is calculated by maximizing Q(Γ, Γ⁽ⁱ⁾) which was obtained from the E-step; that is,

$${\mathsf{\Gamma }}^{(i+1)}=\underset{\mathsf{\Gamma }}{\arg \max }Q(\mathsf{\Gamma },{\mathsf{\Gamma }}^{\left(i\right)}).$$

(3)

The iteration between the E- and M-steps continues until the elements of Γ are less than a given value.

2.3. The Coefficient of Determination

Analysis of variance is an approach for determining the significance and validity of a regression model using variances obtained from the data and model. The coefficient of determination is an analysis of variance approach that seeks to decompose the total variability in the data into various orthogonal components that can then be independently analyzed [23]. For the purposes of analyzing the regression, let the total sum of squares, denoted by TSS, be defined as

$$TSS={\displaystyle \sum _{i=1}^n{(y_i-\overline{y})}^2}$$

(4)

where the real data set is represented as y = <y₁, y₂, …, y_n> and $\overline{y}$ refers to the average of y_i. Let the sum of squares due to regression, SSR, be defined as

$$SSR={\displaystyle \sum _{i=1}^n{({\widehat{y}}_i-\overline{y})}^2}$$

(5)

where ${\widehat{y}}_i$ denotes the predicted value of the regression model for y_i. The coefficient of determination R² represents the ratio of SSR to TSS, that is,

$$R^2=\frac{SSR}{TSS}.$$

(6)

Let the sum of squares due to the error, SSE, be defined as

$$SSE={\displaystyle \sum _{i=1}^n{(y_i-{\widehat{y}}_i)}^2}.$$

(7)

It can be proved that TSS = SSR + SSE [23,28], so R² can also be expressed as

$$R^2=1-\frac{SSE}{TSS}=1-\frac{{\displaystyle \sum _{i=1}^n{\left(y_i-{\widehat{y}}_i\right)}^2}}{{\displaystyle \sum _{i=1}^n{\left(y_i-\overline{y}\right)}^2}}.$$

(8)

3. Development of the Probabilistic Soft Sensor Model

In this section, in order to obtain more accurate KPI estimates, a soft sensor development approach based on maximizing the coefficient of determination is proposed. In addition, the problem of missing data in the training sample set is also considered. In order to more clearly describe the soft sensor development process, Figure 1 shows the modeling flow chart.

3.1. EM Algorithm Handing Missing Data

Let X₁, X₂, …, X_n be a random sample from a p-variate normal population, where X_j = (x_j1, x_j2, …, x_jp), 1 ≤ j ≤ n, so the training sample set X can be written as

$$X=\left(\begin{array}{l}{X}_1\\ X_2\\ ⋮\\ X_n\end{array}\right)=\left[\begin{array}{llll}{x}_{11},& x_{12},& \cdots ,& x_{1p}\\ x_{21},& x_{22},& \cdots ,& x_{2p}\\ ⋮& & \ddots & ⋮\\ x_{n1},& x_{n2},& \cdots ,& x_{np}\end{array}\right].$$

(9)

The basic steps for processing missing data using the EM algorithm are given in [29].

3.1.1. E-Step: Prediction

For each sample X_j containing missing values, X_j = (m_j, a_j), where m_j is the missing value and a_j is the available values. Given the population mean and variance, ${\tilde{\mu }}^i$ and ${\tilde{\Sigma }}^i$, from the ith iteration and a_j, we use the expectation of the conditional normal distribution of m_j as the estimate of the missing value. The (i + 1)th iteration is

$${\tilde{m}}_j^{i+1}=E(m_j|a_j,{\tilde{\mathsf{\mu }}}^i,{\tilde{\sum }}^i)={\tilde{\mathsf{\mu }}}_m^i+{\tilde{\sum }}_{ma}^i{({\tilde{\sum }}_{aa}^i)}^{-1}(a_j-{\tilde{\mathsf{\mu }}}_a^i)$$

(10)

where ${\tilde{\mu }}^i$ is a p × 1 matrix defined as ${\tilde{\mathsf{\mu }}}^i={\left[{\tilde{\mathsf{\mu }}}_m^i,{\tilde{\mathsf{\mu }}}_a^i\right]}^{\prime }$, ${\tilde{\mathsf{\mu }}}_m^i$ is the mean of the missing part, and ${\tilde{\mathsf{\mu }}}_a^i$ is the mean of the available part. In addition, ${\tilde{\Sigma }}^i$ can be written as

$${\tilde{\sum }}^i=\left[\begin{array}{l}{\tilde{\sum }}_{mm}^i{\tilde{\sum }}_{ma}^i\\ {\tilde{\sum }}_{am}^i{\tilde{\sum }}_{aa}^i\end{array}\right].$$

(11)

3.1.2. M-Step: Estimation

We compute the maximum likelihood estimates as follows:

$${\tilde{\mathsf{\mu }}}^{i+1}={\overline{X}}^{i+1}$$

(12)

$${\tilde{\sum }}^{i+1}=\frac{\left(n-1\right)S^{i+1}}{n}$$

(13)

where ${\overline{X}}^{i+1}$ is the mean of the samples and Sⁱ⁺¹ is the sample standard deviation, and they are all sufficient statistics. For a normal population, the importance of sufficient statistics is that the total information about μ and Σ in the data matrix X is contained in $\overline{X}$ and S, regardless of the sample size n. By transforming $\overline{X}$ and S, two new sufficient statistics T₁ and T₂ [29], given by

$$T_1=n\overline{X}$$

(14)

$$T_2=\left(n-1\right)S+n\overline{X}{\overline{X}}^{\prime }$$

(15)

are obtained. Combining Equations (14) and (15) with Equations (12) and (13) gives

$${\tilde{\mathsf{\mu }}}^{i+1}=\frac{T_1^{i+1}}{n}$$

(16)

$${\tilde{\sum }}^{i+1}=\frac{1}{n}{T}_2{}^{i+1}-{\tilde{\mathsf{\mu }}}^{i+1}{({\tilde{\mathsf{\mu }}}^{i+1})}^{\prime }$$

(17)

where

$${\widetilde{m_j{m}_j{}^{\prime }}}^{i+1}=E(m_j{m}_j{}^{\prime }|a_j,{\tilde{\mathsf{\mu }}}^i,{\tilde{\sum }}^i)={\tilde{\sum }}^i{}_{mm}-{\tilde{\sum }}^i{}_{ma}{({\tilde{\sum }}^i{}_{aa})}^{-1}{\tilde{\sum }}^i{}_{am}+{\tilde{m}}_j^{i+1}{\left({\tilde{m}}_j^{i+1}\right)}^{\prime }$$

(18)

$${\widetilde{m_j^{}{a}_j{}^{\prime }}}^{i+1}=E(m_j^{}{a}_j{}^{\prime }|a_j,{\tilde{\mathsf{\mu }}}^i,{\tilde{\sum }}^i)={\tilde{m}}_j^{i+1}{\left(a_j^{}\right)}^{\prime }.$$

(19)

The iteration between the E- and M-steps continues until the elements of $\tilde{\mu }$ and $\tilde{\Sigma }$ are less than a given value. Therefore, the iteration result $\tilde{m}$ is the optimal substitution for the missing values, resulting in a complete training sample set X.

3.2. Soft Sensor Development Approach Based on the Coefficient of Determination Maximization Strategy

For the complete training sample set X obtained from Section 3.1, which can be written as

$$X=\left[\begin{array}{llll}{x}_{11},& x_{12},& \cdots ,& x_{1p}\\ x_{21},& x_{22},& \cdots ,& x_{2p}\\ ⋮& & \ddots & ⋮\\ x_{n1},& x_{n2},& \cdots ,& x_{np}\end{array}\right]$$

(20)

let $\left(x_1,x_2,\cdots x_{p-1}\right)$ denote the secondary variables, and x_p denote the KPI. Our objective is to estimate x_p from $\left(x_1,x_2,\cdots x_{p-1}\right)$.

R² measures the fraction of the total variance in the model explained by the regression with the given variables [23]. The range of R² is [0,1]. Let x_p be the y mentioned in Section 2.3. Then, the coefficient of determination is

$$R^2=1-\frac{{\displaystyle \sum _{i=1}^n{\left(x_{ip}-{\widehat{x}}_{ip}\right)}^2}}{{\displaystyle \sum _{i=1}^n{\left(x_{ip}-{\overline{x}}_p\right)}^2}}.$$

(21)

If the secondary variables in the soft sensor model do not account for the variance of x_p, the estimate of x_ip, denoted ${\widehat{x}}_{ip}$, is exactly equal to the sample mean of x_ip, denoted ${\overline{x}}_{ip}$. In this case, SSR is 0 and SSE equal to TSS, so R² = 0. On the other hand, if $\left(x_{i1},x_{i2},\cdots x_{i(p-1)}\right)$ fully explains the variance of x_ip, for i = 1, 2,…, n, it follows that x_ip = ${\overline{x}}_{ip}$, i.e., each error is zero and SSR = TSS, so R² = 1. In general, R² does not take the extreme values 0 or 1, but instead takes a certain value between the two [28]. For the case where the number of variables, p, is much smaller than the sample number n, the closer R² is to 1, the better the model. Therefore, when the model for the KPI maximizes R², it becomes the best estimate of the KPI, that is,

$$1-\frac{{\displaystyle \sum _{i=1}^n{\left(x_{ip}-{\tilde{x}}_{ip}\right)}^2}}{{\displaystyle \sum _{i=1}^n{\left(x_{ip}-{\overline{x}}_p\right)}^2}}=\max \left[1-\frac{{\displaystyle \sum _{i=1}^n{\left(x_{ip}-K_i\right)}^2}}{{\displaystyle \sum _{i=1}^n{\left(x_{ip}-{\overline{x}}_p\right)}^2}}\right]$$

(22)

where ${\tilde{x}}_{ip}$ is the best estimate of x_ip, and K_i represents all possible estimates of x_ip. Simplifying the above equation gives

$$\frac{{\displaystyle \sum _{i=1}^n{\left(x_{ip}-{\tilde{x}}_{ip}\right)}^2}}{{\displaystyle \sum _{i=1}^n{\left(x_{ip}-{\overline{x}}_p\right)}^2}}=\min \left[\frac{{\displaystyle \sum _{i=1}^n{\left(x_{ip}-K_i\right)}^2}}{{\displaystyle \sum _{i=1}^n{\left(x_{ip}-{\overline{x}}_p\right)}^2}}\right]$$

(23)

where x_ip and ${\overline{x}}_p$ are both computed values. Equation (23) can then be written as

$${\displaystyle \sum _{i=1}^n{\left(x_{ip}-{\tilde{x}}_{ip}\right)}^2}=\min \left[{\displaystyle \sum _{i=1}^n{\left(x_{ip}-K_i\right)}^2}\right].$$

(24)

Multiplying Equation (24) on both sides by n⁻¹ gives

$$\frac{1}{n}{\displaystyle \sum _{i=1}^n{\left(x_{ip}-{\tilde{x}}_{ip}\right)}^2}=\min \left[\frac{1}{n}{\displaystyle \sum _{i=1}^n{\left(x_{ip}-K_i\right)}^2}\right].$$

(25)

Considering that the mathematical expectation of a discrete random variable is

$$E\left(x\right)={\displaystyle \sum _i{x}_i}{p}_i$$

(26)

where x_i represents the ith value of the random variable x and p_i represents its probability, Equation (26) can be expressed as

$$E\left\{{\Vert x_p-{\tilde{x}}_p\Vert }^2\right\}=\min E\left\{{\Vert x_p-K\Vert }^2\right\}$$

(27)

where K denotes all possible estimates of the KPI x_p, and ${\tilde{x}}_p$ represents the best estimate of the KPI when the coefficient of determination R² is maximized. Since x_p is derived from the soft sensor models and secondary variables, the above equation can be written as

$${\tilde{x}}_p=\underset{K}{\arg \min }E\left[{\Vert x_p-K\Vert }^2|\left(x_1,x_2,\cdots x_{p-1}\right)\right].$$

(28)

In order to establish a more direct connection between ${\tilde{x}}_p$ and (x_i1, x_i2, …, x_i(p–1)), the left-hand side of Equation (28) will be simplified further. Firstly, it can be noted that K does not have an impact on the simplification, that is,

$$\begin{array}{cc}& E\left[{\Vert x_p-K\Vert }^2|\left(x_1,x_2,\cdots x_{p-1}\right)\right]\hfill \\ \hfill =& E\left[{\Vert x_p-E\left(x_p|\left(x_1,x_2,\cdots x_{p-1}\right)\right)+E\left(x_p|\left(x_1,x_2,\cdots x_{p-1}\right)\right)-K\Vert }^2|\left(x_1,x_2,\cdots x_{p-1}\right)\right]\hfill \\ \hfill =& E\left[{\Vert x_p-E\left(x_p|\left(x_1,x_2,\cdots x_{p-1}\right)\right)\Vert }^2|\left(x_1,x_2,\cdots x_{p-1}\right)\right]+E\left[{\Vert E\left(x_p|\left(x_1,x_2,\cdots x_{p-1}\right)\right)-K\Vert }^2|\left(x_1,x_2,\cdots x_{p-1}\right)\right]\hfill \\ & +E\left[{\left[x_p-E\left(x_p|\left(x_1,x_2,\cdots x_{p-1}\right)\right)\right]}^T\left[E\left(x_p|\left(x_1,x_2,\cdots x_{p-1}\right)\right)-K\right]|\left(x_1,x_2,\cdots x_{p-1}\right)\right]\hfill \\ & +E\left[{\left[E\left(x_p|\left(x_1,x_2,\cdots x_{p-1}\right)\right)-K\right]}^T\left[x_p-E\left(x_p|\left(x_1,x_2,\cdots x_{p-1}\right)\right)\right]|\left(x_1,x_2,\cdots x_{p-1}\right)\right]\hfill \end{array}$$

(29)

In order to minimize the above equation, the following should hold:

$$K=E\left[x_p|\left(x_1,x_2,\cdots x_{p-1}\right)\right]$$

(30)

which can be rewritten as

$${\tilde{x}}_p=E\left[x_p|\left(x_1,x_2,\cdots x_{p-1}\right)\right].$$

(31)

Furthermore, $E\left[x_p|\left(x_1,x_2,\cdots x_{p-1}\right)\right]$ can be expanded according to the definition of expectation, giving

$$\begin{array}{cc}\hfill {\tilde{x}}_p& =E\left[x_p|\left(x_1,x_2,\cdots x_{p-1}\right)\right]\hfill \\ & ={\displaystyle \int x_{p}p\left[x_p|\left(x_1,x_2,\cdots x_{p-1}\right)\right]d}{x}_p.\hfill \\ & ={\displaystyle \int x_p\frac{p\left(x_1,x_2,\cdots x_{p-1},x_p\right)}{p\left(x_1,x_2,\cdots x_{p-1}\right)}d}{x}_p\hfill \end{array}$$

(32)

Thus, this establishes the basic framework of the probabilistic soft sensor model with KPI optimal estimation.

The next part is to solve the joint probability distribution in the model.

In this paper, GMM is used to approximate the joint probability distribution. Let $p\left(x_e\right)=p\left(x_1,x_2,\cdots x_{p-1}\right)$; that is,

$$p\left(x_e\right)={\displaystyle \sum _{j=1}^M{\alpha }_j}p\left(x_{je}|{\theta }_j\right)$$

(33)

$$p\left(x_e,x_p\right)={\displaystyle \sum _{l=1}^M{\alpha }_l}p\left(x_{le},x_{lp}|{\theta }_l\right).$$

(34)

In order to deduce the specific representation of the KPI optimal estimation ${\tilde{x}}_p$ under the proposed probabilistic soft sensor model, we first introduce Lemma 1.

Lemma 1.

[30] Let G(x; μ, Σ) be a multidimensional normal density function with mean μ and covariance matrix Σ. Let $x^T=\left(x_1^T,x_2^T\right)$, ${\mathsf{\mu }}^T=\left({\mathsf{\mu }}_1^T,{\mathsf{\mu }}_2^T\right)$, and $\mathsf{\Sigma }=\left[\begin{array}{l}{\mathsf{\Sigma }}_{11}{\mathsf{\Sigma }}_{12}\\ {\mathsf{\Sigma }}_{21}{\mathsf{\Sigma }}_{22}\end{array}\right]$; then, the joint probability density is

$$p\left(x\right)=G\left(x_1;{\mathsf{\mu }}_1,{\mathsf{\Sigma }}_{11}\right)G\left(x_2;{\mathsf{\mu }}_{x_2|x_1},{\mathsf{\Sigma }}_{x_2|x_1}\right)$$

(35)

where

$${\mathsf{\mu }}_{x_2|x_1}={\mathsf{\mu }}_2-{\mathsf{\Sigma }}_{21}{\mathsf{\Sigma }}_{11}^{-1}\left({\mathsf{\mu }}_1-x_1\right)$$

(36)

$${\mathsf{\Sigma }}_{x_2|x_1}={\mathsf{\Sigma }}_{22}-{\mathsf{\Sigma }}_{21}{\mathsf{\Sigma }}_{11}^{-1}{\mathsf{\Sigma }}_{12}.$$

(37)

Proof. 

The details of the proof can be found in [30].

Using Lemma 1, it follows that

$$\begin{array}{cc}& p\left(x_{le},x_{lp}\right)\hfill \\ \hfill =& G\left(x_l;{\mathsf{\mu }}_l,{\mathsf{\Sigma }}_l\right)\hfill \\ \hfill =& G\left(x_{le};{\mathsf{\mu }}_{le},{\mathsf{\Sigma }}_{lee}\right)G\left(x_{lp};{\mathsf{\mu }}_{lp|e},{\mathsf{\Sigma }}_{lp|e}\right)\hfill \end{array}$$

(38)

where ${\mathsf{\mu }}_l=\left({\mathsf{\mu }}_{le}^T,{\mathsf{\mu }}_{lp}^T\right)$ and ${\mathsf{\Sigma }}_l=\left[\begin{array}{l}{\mathsf{\Sigma }}_{lee}{\mathsf{\Sigma }}_{lep}\\ {\mathsf{\Sigma }}_{lpe}{\mathsf{\Sigma }}_{lpp}\end{array}\right]$. Therefore, Equations (33) and (34) can be written as

$$p\left(x_e\right)={\displaystyle \sum _{j=1}^M{\alpha }_j}G\left(x_{je};{\mathsf{\mu }}_{je},{\mathsf{\Sigma }}_{jee}\right)$$

(39)

$$p\left(x_e,x_p\right)={\displaystyle \sum _{l=1}^M{\alpha }_l}G\left(x_{le};{\mathsf{\mu }}_{le},{\mathsf{\Sigma }}_{lee}\right)G\left(x_{lp};{\mathsf{\mu }}_{lp|e},{\mathsf{\Sigma }}_{lp|e}\right).$$

(40)

Substituting Equations (39) and (40) into Equation (32) gives

$$\begin{array}{cc}\hfill {\tilde{x}}_p& ={\displaystyle \int x_p\frac{p\left(x_e,x_p\right)}{p\left(x_e\right)}d}{x}_p\hfill \\ & ={\displaystyle \int x_p\frac{{\displaystyle \sum _{l=1}^M{\alpha }_l}G\left(x_{le};{\mathsf{\mu }}_{le},{\mathsf{\Sigma }}_{lee}\right)G\left(x_{lp};{\mathsf{\mu }}_{lp|e},{\mathsf{\Sigma }}_{lp|e}\right)}{{\displaystyle \sum _{j=1}^M{\alpha }_j}G\left(x_{je};{\mathsf{\mu }}_{je},{\mathsf{\Sigma }}_{jee}\right)}d}{x}_p.\hfill \end{array}$$

(41)

Extracting the sum in the numerator to outside the integral gives

$${\tilde{x}}_p={\displaystyle \sum _{l=1}^M{\displaystyle \int x_p\frac{{\alpha }_{l}G\left(x_{le};{\mathsf{\mu }}_{le},{\mathsf{\Sigma }}_{lee}\right)G\left(x_{lp};{\mathsf{\mu }}_{lp|e},{\mathsf{\Sigma }}_{lp|e}\right)}{{\displaystyle \sum _{j=1}^M{\alpha }_j}G\left(x_{je};{\mathsf{\mu }}_{je},{\mathsf{\Sigma }}_{jee}\right)}d}{x}_p}.$$

(42)

In order to make the derivation more concise, the positions of some factors in the integral are changed as follows:

$$\begin{array}{cc}\hfill {\tilde{x}}_p& ={\displaystyle \sum _{l=1}^M{\displaystyle \int \frac{{\alpha }_{l}G\left(x_{le};{\mathsf{\mu }}_{le},{\mathsf{\Sigma }}_{lee}\right)}{{\displaystyle \sum _{j=1}^M{\alpha }_j}G\left(x_{je};{\mathsf{\mu }}_{je},{\mathsf{\Sigma }}_{jee}\right)}{x}_{p}G\left(x_{lp};{\mathsf{\mu }}_{lp|e},{\mathsf{\Sigma }}_{lp|e}\right)d}{x}_p}\hfill \\ & ={\displaystyle \sum _{l=1}^M\frac{{\alpha }_{l}G\left(x_{le};{\mathsf{\mu }}_{le},{\mathsf{\Sigma }}_{lee}\right)}{{\displaystyle \sum _{j=1}^M{\alpha }_j}G\left(x_{je};{\mathsf{\mu }}_{je},{\mathsf{\Sigma }}_{jee}\right)}{\displaystyle \int x_{p}G\left(x_{lp};{\mathsf{\mu }}_{lp|e},{\mathsf{\Sigma }}_{lp|e}\right)d}{x}_p}.\hfill \end{array}$$

(43)

When the integral part is the conditional expectation, the above equation can be simplified to

$${\tilde{x}}_p={\displaystyle \sum _{l=1}^M\frac{{\alpha }_{l}G\left(x_{le};{\mathsf{\mu }}_{le},{\mathsf{\Sigma }}_{lee}\right)}{{\displaystyle \sum _{j=1}^M{\alpha }_j}G\left(x_{je};{\mathsf{\mu }}_{je},{\mathsf{\Sigma }}_{jee}\right)}{\mathsf{\mu }}_{lp|e}}.$$

(44)

Therefore, the detailed soft sensor model expression of the KPI optimal estimation is obtained.

In this paper, unknown parameters in the model are estimated using the EM algorithm. The iterative equations of the EM algorithm for estimating the GMM parameters are [31]

$${\mathsf{\mu }}_l^{(i+1)}=\frac{{\displaystyle \sum _{j=1}^n{\gamma }_{jl}^{(i+1)}{X}_j}}{{\displaystyle \sum _{j=1}^n{\gamma }_{jl}^{(i+1)}}},{\mathsf{\Sigma }}_l^{(i+1)}=\frac{{\displaystyle \sum _{j=1}^n{\gamma }_{jl}^{(i+1)}{\left(X_j-{\mathsf{\mu }}^{(i)}\right)}^2}}{{\displaystyle \sum _{j=1}^n{\gamma }_{jl}^{(i+1)}}},{\alpha }_l^{(i+1)}=\frac{{\displaystyle \sum _{j=1}^n{\gamma }_{jl}^{(i+1)}}}{n}$$

(45)

where γ_jl represents the responsivity of the mixed component l on the training sample data X_j. It can be written as

$${\gamma }_{jl}^{(i+1)}=\frac{{\alpha }_{l}p\left(X_j|{\theta }_l\right)}{{\displaystyle \sum _{l=1}^M{\alpha }_{l}p\left(X_j|{\theta }_l\right)}}.$$

(46)

Consequently, the above steps give the GMM parameters, and the KPI optimal estimate ${\tilde{x}}_p$ follows.

4. Case Study

In this section, the effectiveness and feasibility of the proposed soft sensor model approach based on maximizing the coefficient of determination are evaluated through an industrial aluminum electrolytic production process. To show the advantages of the probabilistic soft sensor framework, the estimations are compared with the real values. For performance evaluation, the root-mean-squared error (RMSE) index is used.

4.1. Soft Sensor Development for Industrial Aluminum Electrolytic Process

Aluminum is widely used in construction and electrical industries [32]. The main method currently chosen for smelting aluminum plants is the cryolite–alumina molten salt electrolysis process, in which the electrochemical reaction process takes place in an electrolytic cell. Figure 2 shows the internal structure of the electrolytic cell.

Molten cryolite is a solvent in which aluminum oxide is dissolved as a solute, forming a melt with good electrical conductivity. Carbon materials are used as cathodes and anodes, and a direct current is passed through them. The thermal energy of the direct current is used to melt the cryolite and maintain a constant electrolysis temperature. Furthermore, the electrochemical reaction occurs between the two electrodes, where the product at the cathode is aluminum liquid, and carbon dioxide and other gases are generated at the anode. The chemical reaction of the electrolytic process is

2Al₂O₃ + 3C → 4Al + 3CO_2↑.

The chemical reaction can produce gases other than carbon dioxide and carbon monoxide, as well as fluorocarbon gases. The gas purifying device uses alumina and fluorine generated in the mixed gas to produce fluorinated alumina, and the fluorinated alumina is then recycled to the electrolytic cell for chemical reaction. Figure 3 shows the process flow diagram of the aluminum electrolysis process.

The main control goal of the aluminum electrolysis process is to keep the alumina concentration in the electrolysis cell stable within a certain range, preferably between 1.5% and 3.5% [33]. The control of alumina concentration relates to energy consumption and economic benefits of the aluminum electrolytic production process. On one hand, when the alumina concentration is too low, an additional chemical reaction occurs at the anode, which can easily cause a sudden rise in the cell voltage and the energy balance of the cell is destroyed. On the other hand, when the concentration reaches saturation, if the feeder continues to add alumina at the time, the raw material will be deposited at the bottom of the cell, so that the resistance increases and the current efficiency becomes low. Therefore, it is necessary to keep the alumina concentration in the proper range.

In soft sensor development for the aluminum electrolytic process, the measurable variables, the voltage x₁ between the two electrodes obtained by the first voltage measuring instrument; the anode conductor current x₂; the voltage x₃ between the two electrodes obtained by the second voltage measuring instrument; and the alumina concentration x₄ provided by an electrochemical analyzer, were selected as the secondary variables. The interelectrode voltage refers to the voltage between the anode guide and the corresponding cathode steel bar. The alumina concentration y provided by the laboratory is the primary variable for the model. Figure 4 shows a diagram of the process measurement system.

The variables x₁(k), x₂(k), x₃(k), x₄(k), and y(k) form the joint probability distribution

$$p\left(x(k)\right)=p(x_1(k),x_2(k),x_3(k),x_4(k),y(k)).$$

(47)

The soft sensor was then developed according to the process described in Section 3 of this paper. It is assumed that M = 2.

4.2. Experimental Results

4.2.1. EM Algorithm and Missing Values

We took 600 complete data groups from the training sample set, and deleted 10%, 20%, or 30% of the alumina concentration variable data. Then, the mean substitution method, the regression interpolation method, and the EM algorithm were used to process the sample set with missing values.

Table 1. Comparison of three data interpolation methods for a 10% missing rate.

	Mean Substitution Method	Regression Interpolation Method	EM Algorithm	Real Value
Mean	2.4133	2.4225	2.4225	2.4259
RMSE	0.0867	0.4209	0.0698	0

,

Table 2. Comparison of three data interpolation methods for a 20% missing rate.

	Mean Substitution Method	Regression Interpolation Method	EM Algorithm	Real Value
Mean	2.4139	2.4217	2.4215	2.4259
RMSE	0.1451	0.4075	0.1361	0

and

Table 3. Comparison of three data interpolation methods for a 30% missing rate.

	Mean Substitution Method	Regression Interpolation Method	EM Algorithm	Real Value
Mean	2.4140	2.4204	2.41198	2.4259
RMSE	0.1700	0.4068		0

show the mean and RMSE of the alumina concentration sample set for the three method simulations for missing ratios of 10%, 20%, and 30%.

First, comparing the mean value, we can see from the above tables that the means of the regression interpolation method and the EM data interpolation method are closer to the mean of the real value set, and the mean substitution method is less effective. Obviously, the RMSE of the EM data interpolation method is much smaller than that of the regression interpolation method. Therefore, the accuracy and effectiveness of the EM data interpolation method in processing missing values is verified. Further, if there is a problem with missing values in the practical industrial process, the EM algorithm can be selected for data interpolation.

4.2.2. Experimental Results of the Soft Sensor Model Based on Maximizing the Coefficient of Determination

In order to verify the feasibility of the proposed approach, a test sample set was used to validate the designed soft sensor model. The test sample set was divided into four subsets of 100 samples. The actual alumina concentration measurement obtained from the laboratory was compared with the output of the soft sensor model to acquire an estimated performance evaluation of the model. The results are shown in Figure 5. Figure 5a–d show the estimated alumina concentrations based on the first, second, third, and fourth test subsets, respectively.

Table 4. The RMSE values of the four test subsets.

Test Subset	RMSE
First	0.0231
Second	0.0145
Third	0.0209
Fourth	0.0155

shows the root-mean-square errors (RMSE) of the four test subsets. It can be seen that, overall, the soft sensor model based on maximizing the coefficient of determination can accurately track the overall trends in the process. The alumina concentration output by the model is approximately the same as the actual laboratory measurement.

4.2.3. Comparison with BP and LSSVM

The backpropagation (BP) neural network and the least-squares, support vector machine (LSSVM) model were applied to the test sample set, and the first test subset was used for performance comparison. The parameters of the comparison algorithms were determined as follows: The number of hidden layer nodes in the BP neural network model was 100 and the activation function of the hidden layer was a sigmoid [34]. The kernel function of the LSSVM model was the radial basis function (RBF), and the kernel parameter and regular parameter were 1 and 20, respectively [34]. For each model, the number of secondary variables was 4, and the number of primary variables was 1. It could be seen that the two comparison models need different parameters in order to achieve an accurate estimation performance, while this is not necessary for the soft sensor model based on maximizing the coefficient of determination. The estimated results are shown in Figure 6 and Figure 7. Figure 6 shows the estimated values of the soft sensor based on the BP neural network for the first test subset, and Figure 7 shows the estimated values of the soft sensor based on the LSSVM for the first test subset. It can be seen from Figure 6 that the soft sensor based on a BP neural network can roughly follow the trend of the laboratory measurements, but the error is still large at many points. It can be seen from Figure 7 that the overall performance of the soft sensor based on LSSVM is better than that based on a BP neural network, but compared with Figure 5a, it is obvious that the estimation of some extreme points is not as accurate as that given by the soft sensor based on maximizing the coefficient of determination.

Figure 8, Figure 9 and Figure 10 show the soft sensor estimates based on different modelling methods as a function of the laboratory measurements. The green circles show the BP neural network model; the purple circles the LSSVM model; and the red circles the proposed coefficient of determination maximization model. In the ideal case, the circles should lie on the blue y = x line. In practice, deviations from this behavior can provide information about the accuracy of the models. The BP neural network soft sensor produces a soft sensor system that has a consistent bias, since the values are consistently located above the y = x line. Furthermore, the bias in the LSSVM soft sensor model is smaller, but there also seems to be a calibration issue, since the data does not lie parallel to the y = x line. Finally, the proposed model has the smallest deviations and the most ideal performance.

To better illustrate the performance of the proposed soft sensor model,

Table 5. The comparison of the RMSE between the three modelling methods.

Method	RMSE
BP neural network	0.0616
LSSVM	0.0431
Maximizing the Coefficient of Determination	0.0231

shows the RMSE values for the different methods. As can be seen from Table 5, the RMSE of the proposed method is smallest, which means that the estimation effect of the proposed model is better than those of the BP neural network model and the LSSVM model.

5. Conclusions

In this paper, a new KPI estimation method for probabilistic soft sensor development is proposed based on maximizing the coefficient of determination. The joint probability distribution in the probability model is approximated using GMM, while the EM algorithm is used to estimate the GMM parameters. In addition to providing accurate, real-time estimates of the KPIs, this paper also considers the missing values that training sample sets often face and uses the EM algorithm for processing. The resulting soft sensor design method was tested on a case study of the alumina extraction process, which shows that the proposed method can provide alumina concentration estimations that are consistent with the actual measurements obtained from laboratory tests. Future work will focus on applying the proposed soft sensor development approach to solving various problems such as dealing with dynamic, non-Gaussian, or batch processes.