Kernel-Based Multivariate Nonparametric CUSUM Multi-Chart for Detection of Abrupt Changes

Abstract

In many cases, it is difficult to obtain precise distributional information on multivariate sequences. Therefore, there is a need to propose nonparametric methods for monitoring multivariate sequences. This article discusses the multivariate change detection problem and utilizes the kernel function as the statistic to construct the nonparametric Multivariate Cumulative Sum multi-chart, under the assumption that there is prior information about the abrupt changes. Through theoretical and numerical analysis, we show that the proposed control chart is more effective compared to other existing control charts. The good monitoring effect of this method demonstrates a strong potential for application.

1. Introduction

Statistical Process Control (SPC) is an important problem in numerous applications. The development of science and the increase in data storage have led to an increasing need for Multivariate Statistical Process Control (MSPC) [1,2]. The purpose of change detection is to raise an alarm as soon as possible when a change occurs. In order to evaluate the detectability of the control chart, we usually use two kinds of average run lengths ($ARLs$): one is in-control (IC) $ARL$, denoted by $AR{L}_0$, and the other is out-of-control (OC) $ARL$, denoted by $AR{L}_1$. With the same $AR{L}_0$, a control chart with a smaller $AR{L}_1$ is better and can detect changes more effectively.

In the literature, many researchers focus on this area and propose various methods to detect multivariate variables. Multivariate Statistical Process Control techniques were established by Hotelling in his pioneering paper in 1947 [3], which monitored the mean vector of multiple quality variables following a multivariate normal distribution. Subsequently, the field has attracted many experts to conduct related research and many different multivariate control charts have been proposed. In the literature, multivariate control charts can be classified into two categories according to the presence or absence of the assumption of data distribution. The first category with a distribution assumption is parametric. These include the Multivariate Exponentially Weighted Moving Average control chart (MEWMA [4,5]), Multivariate Cumulative Sum control chart (MCUSUM [6,7]) and other control charts, such as [8,9,10], which use different strategies to construct control charts to detect multivariate variables. These charts typically require the distribution of the observed data to obey a parametric (usually normal) form. However, in most cases, we lack the distributional information, or we do not have enough samples to estimate the exact distribution. The second category is nonparametric without distribution assumption, i.e., distribution-free control charts. Unfortunately, unlike many univariate nonparametric charts, it is challenging to design a distribution-free MSPC scheme based on conventional construction [11]. There are many multivariate nonparametric control charts proposed. Refs. [12,13] discussed multivariate process control in the context of multivariate data. Ref. [14] proposed general and multiplicative nonparametric ratio models for data envelopment analysis problems with interval data. Ref. [15] provided a variable sampling interval (VSI) Shewhart $\overline{X}$ chart. Ref. [2] proposed a multivariate sign EWMA chart without distribution assumption, which requires some in-control observations. Other nonparametric multivariate control charts such as [16,17,18] also tried to use different methods to detect changes without parametric assumption.

In this paper, we develop a nonparametric or distribution-free control chart for monitoring multivariate data. Our approach is different from the existing nonparametric SPC methods in the sense that we apply the kernel function to construct the CUSUM multi-chart, that is, we construct a CUSUM-type control chart that does not require the distributional information of the multivariate data. Meanwhile, the method proposed in this paper solves three challenges in multivariate change detection:

• The first challenge is as follows: It is known that the traditional CUSUM control chart needs both pre-change and post-change information [7]. However, in reality, for online detection, we usually do not know the post-change information, we only have some pre-change observations. In order to solve this challenge, we construct the nonparametric multivariate CUSUM multi-chart. The idea of the multi-chart approach originates from [19], which shows that the multi-chart approach can detect different types of changes.
• The second challenge is how to capture important features when constructing a control chart for multivariate data. It is known that for multivariate data, such as when the dimension is $p=30$, which is a relatively large number, statistics like Hotelling’s $T^2$ may contain errors and trigger false alarms in online change detection [3]. In order to overcome this challenge, we utilize the kernel function to capture information for the multivariate data.
• The last challenge is the amount of historical pre-change observations; we may not have too many historical pre-change observations, that is, the amount of historical pre-change observations required for the method proposed in this paper does not need to be very large, making this method promising for application. For nonparametric control charts, historical pre-change observations are required. However, if the number of observations needed is excessively large, the loss outweighs the gain, and in many application scenarios, it is not achievable [20].

From the above statements, we can conclude that the kernel-based multivariate nonparametric CUSUM multi-chart proposed in this paper can overcome the aforementioned three challenges. We also present the theoretical results and simulation results of the control chart proposed in this paper.

The remainder of this paper is organized as follows: In Section 2, we first present a brief review of the reproducing kernel Hilbert space (RKHS) and then introduce the kernel-based multivariate nonparametric CUSUM multi-chart proposed in this paper, along with some theoretical properties. Section 3 uses the simulated data to present that the proposed kernel-based multivariate nonparametric CUSUM multi-chart has a better performance compared with other existing control charts. The conclusion and recommendations are given in Section 4. The proofs of the theoretical results are listed in Appendix A.

2. Kernel-Based Multivariate Nonparametric CUSUM Multi-Chart

In this section, we propose the kernel-based CUSUM multi-chart for the multivariate observation sequences. To illustrate this model in detail, we first briefly review the reproducing kernel Hilbert space (RKHS). This involves providing the definition of the kernel function and the statistics used for change detection. With the kernel function, we define the nonparametric CUSUM multi-chart and obtain the theoretical properties to ensure the performance of this model.

It is known that for a traditional univariate CUSUM control chart, we need to know the pre-change and post-change distributions to define the statistics used in the control chart. However, the post-change information is usually unknown in reality, as is the distribution information. So, in this section, the kernel-based CUSUM multi-chart can solve the above problem when both the post-change information and the distribution information are unknown.

2.1. Examples of Kernels

In this section, we construct the CUSUM multi-chart based on the kernel function. Here, we give some examples of kernels that are widely used, as follows:

• When $\mathcal{X}={\mathcal{R}}^d$, $k^{lin}(x,y)=<x,y{>}_{{\mathcal{R}}^d}$ defines the linear kernel. When $d=1$, the KCP (kernel change-point) then coincides with the algorithm proposed in [21].
• When $\mathcal{X}={\mathcal{R}}^d$, $k_h^G(x,y)={exp[-||x-y||}^2/\left(2h^2\right)]$ defines the Gaussian kernel with bandwidth $h>0$, which is used in the experiments in Section 3.
• When $\mathcal{X}={\mathcal{R}}^d$, $k_h^L(x,y)=exp[-||x-y||/h]$ defines the Laplace kernel with bandwidth $h>0$, which is used in the experiments in Section 3.
• When $\mathcal{X}={\mathcal{R}}^d$, $k_h^e(x,y)=exp(<x,y{>}_{{\mathcal{R}}^d}/h)$ defines the exponential kernel with bandwidth $h>0$. Note that, unlike Gaussian and Laplace kernels, the exponential kernel is not translation-invariant.

Classical kernels can be found in the books by [22], with the advantageous properties of the Gaussian kernel and Laplace kernel; we use these two kernels in the simulations and compare the results with other online detection methods. Kernel change-point methods can be used in offline monitoring problems and online monitoring problems. Here, we consider online change detection. So, we have pre-change information through historical pre-change observations, but the exact post-change information is usually unknown.

2.2. Kernel Change-Point Online Detection Method

Let $X_{-m+1},X_{-m+2},\dots ,X_0,X_1,X_2,X_3,\dots ,X_N$ be the multivariate observation sequence. And, $X_i,-m+1\le i\le N\in {\mathcal{R}}^p$ are all p-dimensional vectors. We assume that $X_{-m+1}$, $X_{-m+2}$, …, $X_0$ are historical pre-change observations, and the change-point is defined as $\tau$ ($1\le \tau \le N$). The observation sequence changes from an in-control to an out-of-control state after the change-point $\tau$. From the m historical pre-change observations, we can estimate the pre-change mean vector denoted as ${\widehat{\mu }}_0\in {\mathcal{R}}^p$. While the post-change mean vector, denoted by ${\mu }_1\in {\mathcal{R}}^p$, usually is not known in online change detection, we can obtain the possible boundary of the post-change mean vector, denoted by D, based on empirical knowledge or prior information. With the possible boundary D, we can define the c reference post-change mean vectors ${\mu }_1^k,1\le k\le c$ to construct multiple control charts to detect different types of changes.

With the kernel function, estimated pre-change mean vector ${\widehat{\mu }}_0$, and possible post-change mean vectors ${\mu }_1^k$, where $1\le k\le c$, we can construct the kernel-based CUSUM multi-chart. The kernel-based CUSUM multi-chart is defined as follows: For fixed $c\ge 2$,

$$\begin{array}{cc}\hfill T_{MC}& ={\displaystyle \underset{1\le k\le c}{min}}{T}_k,\hfill \\ \hfill T_k& =min\left\{1\le n\le N:{\displaystyle \underset{1\le j\le n}{max}}{\displaystyle \sum _{i=j}^n}ln\frac{k(X_i,{\mu }_1^k)}{k(X_i,{\widehat{\mu }}_0)}>d_n^k\right\},\hfill \end{array}$$

(1)

where $d_n^k\in \mathcal{R}$ is called the control limit. If an abrupt change occurs, we warn as soon as the statistic exceeds the control limit. So, it is important to study the statistical properties of $T_{MC}$. For p-dimensional observations, possible changes may occur in each dimension. Suppose that in each dimension we give l pre-specified known reference values, then $c=l^p$. However, in reality, the changes usually only occur in several dimensions, and in the simulation, we consider the changes only occur in selected $[p/5]$ dimensions.

The traditional probability likelihood ratio function is replaced by the kernel ratio function in (1). Under a nonparametric assumption, the CUSUM-type control chart can not use the probability likelihood ratio function. Therefore, we aim to find an alternative function to replace the probability function. The reasons why we consider the kernel function are listed as follows:

• The kernel function contains a lot of information, especially in high dimensional cases, and is commonly used in classification problems [23]. While change detection is a very special “classification” problem, in change detection, we need to classify the data as either in-control or out-of-control data. Similarly, can we apply the kernel function in change detection?
• Actually, the kernel function is widely used in the literature in online and offline change detection problems, as demonstrated in [24,25]. Can we naturally apply this function in a CUSUM-type control chart?

For the two reasons mentioned above, we apply the kernel function to replace the probability function and construct the CUSUM multi-chart to solve the problem that the shift is unknown before detection.

Similarly, for the kernel-based CUSUM multi-chart defined in this section, in order to evaluate the detection power of any kernel-based test T with $AR{L}_0\left(T\right)={\mathbf{E}}_0\left(T\right)=\gamma$, we also propose an index called $KCPI$ (Kernel Control Chart Performance Index), as in [19], which is as follows:

$$KCPI\left(T\right)=exp\{-{\int }_{D}w\left({\mu }_1\right)\frac{AR{L}_1\left(T\right)-AR{L}_1^{*}}{AR{L}_1^{*}}d{\mu }_1\},$$

(2)

where

$$AR{L}_1\left(T\right)={\mathbf{E}}_1\left(T\right).$$

The definition of $ARL$ is the average value of run length T. Since the state is divided into in-control and out-of-control states, there are two $ARLs$. ${\mathbf{E}}_0$ means the mathematical expectation of the in-control state, and the in-control $ARL$ is denoted by $AR{L}_0={\mathbf{E}}_0\left(T\right)$. Similarly, ${\mathbf{E}}_1$ stands for the mathematical expectation of the out-of-control state; the out-of-control $ARL$ is denoted by $AR{L}_1={\mathbf{E}}_1\left(T\right)$. $AR{L}_1^{*}$ is the lower bound and $w\left({\mu }_1\right)$ is the weight of the unknown post-change mean vector ${\mu }_1$, that is, we do not know the exact value of ${\mu }_1$, but from engineering knowledge, we know the possible boundary of ${\mu }_1$ denoted by D and the weight function denoted by $w\left({\mu }_1\right)$. If we have no additional information about the unknown ${\mu }_1$, the weight function is $w\left({\mu }_1\right)=1/{\int }_{D}d{\mu }_1$.

Since the purpose of change detection is to provide an alert as soon as possible in the case of an abrupt change, a smaller value of $AR{L}_1$ represents an earlier alarm. For comparison purposes, we usually need a benchmark, i.e., a common $AR{L}_0$ for different control charts. So, in general, in order to compare the performance of several control charts, we ensure they have a common $AR{L}_0$ and then compare the $AR{L}_1$ of the control charts. The smaller the value of $AR{L}_1$, the better the performance of the control chart, which is the essence of the definition of the $KCPI$. If $KCPI\left(T_{MC}\right)$ is larger, the performance of the kernel-based CUSUM multi-chart is better. So, finding the optimal design of the kernel-based CUSUM multi-chart involves finding the maximum value of $KCPI\left(T_{MC}\right)$. Under the multivariate observation cases, if finding the maximum value of $KCPI\left(T_{MC}\right)$ is hard, we can choose stochastic reference values of post-change ${\mu }_1^k,1\le k\le c$, as long as the corresponding $KCPI$ is large.

2.3. Theoretical Properties of Kernel-Based CUSUM Multi-Chart

In this section, we discuss the theoretical properties of the proposed kernel-based CUSUM multi-chart to ensure the performance of change detection.

Similarly, as in Theorem 3 of [26,27], for a large $\gamma$, we can obtain the lower bound $AR{L}_1^{*}={\left[{\mathbf{E}}_1\left(T\right)\right]}^{*}$ with $AR{L}_0\left(T\right)={\mathbf{E}}_0\left(T\right)=\gamma$, as follows:

$${\left[{\mathbf{E}}_1\left(T\right)\right]}^{*}\sim \frac{ln\gamma }{KI({\mu }_1,{\widehat{\mu }}_0)},$$

(3)

where

$$KI({\mu }_1,{\widehat{\mu }}_0)={\mathbf{E}}_1\left[ln\frac{k(X,{\mu }_1)}{k(X,{\widehat{\mu }}_0)}\right].$$

(4)

For a large $N,\gamma$, in order to obtain the asymptotic $AR{L}_1$ of the kernel-based CUSUM multi-chart, we first divide the region D into several disjointed subsets by using the reference post-change ${\mu }_1^k,1\le k\le c$, according to the kernel-based Kullback–Leibler information distance in (4).

Let

$$D_k=\left\{{\mu }_1\in D:KI({\mu }_1,{\mu }_1^k)\le \underset{l\ne k}{min}KI({\mu }_1,{\mu }_1^l)\right\}$$

for $1\le k\le c$. Thus, we have a disjointed division of the region D, with $D_k,1\le k\le c$, and then,

$$KCPI\left(T_{MC}\right)=exp\{-\sum _{k=1}^c{\int }_{D_k}w\left({\mu }_1\right)\frac{AR{L}_1\left(T_{MC}\right)-AR{L}_1^{*}}{AR{L}_1^{*}}d{\mu }_1\}.$$

(5)

The definition of the kernel-based inner Kullback–Leibler information distance is as follows:

$$\begin{array}{cc}\hfill KI({\mu }_1,{\mu }_1^k,{\widehat{\mu }}_0)& ={\mathbf{E}}_1\left[ln\frac{k(X,{\mu }_1^k)}{k(X,{\widehat{\mu }}_0)}\right]\hfill \\ \hfill & ={\mathbf{E}}_1\left[ln\frac{k(X,{\mu }_1)}{k(X,{\widehat{\mu }}_0)}\right]-{\mathbf{E}}_1\left[ln\frac{k(X,{\mu }_1)}{k(X,{\mu }_1^k)}\right]\hfill \\ \hfill & =KI({\mu }_1,{\widehat{\mu }}_0)-KI({\mu }_1,{\mu }_1^k).\hfill \end{array}$$

(6)

The last equation holds with the definition of $KI({\mu }_1,{\widehat{\mu }}_0)$ and $KI({\mu }_1,{\mu }_1^k)$ in (4). Here, we assume that the reference ${\mu }_1^k,1\le k\le c$ satisfies the following conditions:

(1)

$KI({\mu }_1,{\mu }_1^k,{\widehat{\mu }}_0)>0$ if ${\mu }_1\in D_k$ for $1\le k\le c$.

(2)

$KI({\mu }_1,{\widehat{\mu }}_0)\ge 0$ for ${\mu }_1\in D$.

The first condition means that if ${\mu }_1\in D_k$, then $KI({\mu }_1,{\widehat{\mu }}_0)>KI({\mu }_1,{\mu }_1^k)$, which is consistent with the intuitive reality. We can interpret that the “distance” from ${\mu }_1$ to ${\mu }^k$ is smaller than the distance from ${\mu }_1$ to ${\widehat{\mu }}_0$. The reference post-change ${\mu }_1^k$ should be designed to be closer to the real but unknown post-change ${\mu }_1$. The second condition ensures that the kernel-based Kullback–Leibler information distance is non-negative, which imposes constraints on the selection of different kernel functions. So, in the simulation, we choose the Gaussian kernel and Laplace kernel; these two kernel functions all satisfy the above two conditions. In order to give a clear form of (5), we set

$$d_{lim}^k=\underset{n\to \infty }{lim}{d}_n^k.$$

By using a similar method as in [28], we can prove the following theorems. Theorem 1 shows the asymptotic results regarding $AR{L}_1$ for a large $N,\gamma$.

Theorem 1.

Let $T_k,1\le k\le c$ have a common $AR{L}_0={\mathit{E}}_0\left(T_k\right)=\gamma$. If ${\mu }_1\in D_k$, then, a.s.,

$$\begin{array}{cc}\hfill & \frac{T_k}{T_l}\le \frac{max(0,KI({\mu }_1,{\mu }_1^l,{\widehat{\mu }}_0)}{KI({\mu }_1,{\mu }_1^k,{\widehat{\mu }}_0)},\hfill \\ \hfill & T_k<T_l,\hfill \\ \hfill & \frac{T_{MC}}{d_{lim}^k}=\frac{T_k}{d_{lim}^k}+o\left(1\right)\to \frac{1}{KI({\mu }_1,{\mu }_1^k,{\widehat{\mu }}_0)}\hfill \end{array}$$

(7)

for $\tau =1,k\ne l$, and

$$\begin{array}{cc}\hfill & {\mathit{E}}_1\left[\frac{T_{MC}}{d_{lim}^k}\right]={\mathit{E}}_1\left[\frac{T_k}{d_{lim}^k}\right]+o\left(1\right)=\frac{1}{KI({\mu }_1,{\mu }_1^k,{\widehat{\mu }}_0)}(1+o\left(1\right))\hfill \end{array}$$

(8)

for large $N,\gamma$.

By introducing the kernel-based Kullback–Leibler information distance in (4) and the kernel-based inner Kullback–Leibler information distance in (6), we can obtain the statistical properties of $T_{MC}$. For the kernel-based CUSUM multi-chart, in order to compare their $AR{L}_{1}s$, there are two common $AR{L}_{0}s$, ${\gamma }^{\prime }$ and $\gamma$, respectively, for $T_k$ and $T_{MC}^{\prime }$. The dynamic control limits $d_n^{\prime k},1\le k\le c$ are for ${\gamma }^{\prime }$, and $d_n^k,1\le k\le c$ are for $\gamma$, and the inequality $d_{lim}^{\prime k}>d_{lim}^k$ holds. Writing $T_k^{\prime }=T\left(d_n^{\prime k}\right)$ for short, then

$${\mathbf{E}}_0\left[T_k^{\prime }\right]={\gamma }^{\prime }>{\mathbf{E}}_0\left[T_k\right]=\gamma ={\mathbf{E}}_0\left(T_{MC}^{\prime }\right)$$

(9)

for $1\le k\le c$, where $T_{MC}^{\prime }={min}_{1\le k\le c}\left\{T_k^{\prime }\right\}$.

So, from Theorem 1, we have

$$AR{L}_1\left(T_{MC}^{\prime }\right)\to \frac{d_{lim}^{\prime k}}{KI({\mu }_1,{\mu }_1^k,{\widehat{\mu }}_0)}a.s.$$

And, from (3),

$$AR{L}_1^{*}\to \frac{ln\gamma }{KI({\mu }_1,{\widehat{\mu }}_0)}a.s.$$

Then, $KCPI\left(T_{MC}\right)$ can have a simple form for a large N and $\gamma$:

$$\begin{array}{cc}\hfill KCPI\left(T_{MC}\right)& =KCPI({\mu }_1^k,1\le k\le c)=exp\{-{\int }_{D}w\left({\mu }_1\right)\frac{AR{L}_1\left(T\right)-AR{L}_1^{*}}{AR{L}_1^{*}}d{\mu }_1\}\hfill \\ \hfill & =exp\left\{1-{\int }_{D}w\left({\mu }_1\right)\frac{AR{L}_1\left(T\right)}{AR{L}_1^{*}}d{\mu }_1\right\}\hfill \\ \hfill & =exp\left\{1-{\displaystyle \sum _k}{\int }_{D_k}w\left({\mu }_1\right)\frac{KI({\mu }_1,{\widehat{\mu }}_0)}{KI({\mu }_1,{\widehat{\mu }}_0)-KI({\mu }_1,{\mu }_1^k)}d{\mu }_1\right\}.\hfill \end{array}$$

(10)

Then, we give a theorem about the relationship between the kernel-based CUSUM multi-chart and its constituent chart. Indeed, Theorem 2 shows that the performance of the kernel-based CUSUM multi-chart is better than that of its constituent charts in detecting an unknown abrupt change. That is to say, it makes sense to use the kernel-based CUSUM multi-chart; the kernel-based CUSUM multi-chart has better performance than a single kernel-based CUSUM chart in detecting unknown abrupt changes.

Theorem 2.

$p_l,1\le l\le c$ are positive numbers and satisfy

$$\sum _{1\le l\le c}{p}_l=1.$$

If condition (7) holds for $T_{MC}^{\prime }$ and for ${\mu }_1\in D_k$, there is ${\mu }_1^l$, $k\ne l$ such that $KI({\mu }_1,{\mu }_1^l,{\widehat{\mu }}_0)<KI({\mu }_1,{\mu }_1^k,{\widehat{\mu }}_0)$, and then for every ${\mu }_1\in D$,

$$\sum _{1\le l\le c}{p}_{l}AR{L}_1\left(T_l\right)>AR{L}_1\left(T_{MC}^{\prime }\right)$$

for large $N,\gamma$.

The theorems above illustrate the beneficial properties of our proposed kernel-based CUSUM multi-chart.

3. Comparison and Analysis of Simulation Results

In this section, we present simulation results to demonstrate the performance of the proposed kernel-based CUSUM multi-chart. And, we choose the commonly used multivariate normal distribution, multivariate t distribution, and multivariate Gamma distribution as the simulation examples. For comparison, we use different control charts to compute the out-of-control average run length $AR{L}_1$, under the condition that the in-control average run length $AR{L}_0$ is the same. For detection problems, we expect to detect changes promptly as soon as changes occur. Hence, with the same in-control average run length ($AR{L}_0$), the smaller the value of out-of-control average run length ($AR{L}_1$), the better the performance of the detection.

In the simulation, in order to evaluate the performance of change detection for different shifts, the criterion is as follows:

$$\begin{array}{cc}\hfill KCPI\left(T\right)& =exp\{-{\int }_{D}w\left({\mu }_1\right)\frac{AR{L}_1\left(T\right)-AR{L}_1^{*}}{AR{L}_1^{*}}d{\mu }_1\}\hfill \\ & \propto -{\int }_{D}w\left({\mu }_1\right)\frac{AR{L}_1\left(T\right)-AR{L}_1^{*}}{AR{L}_1^{*}}d{\mu }_1\hfill \\ & \propto -{\int }_D\frac{AR{L}_1\left(T\right)-AR{L}_1^{*}}{AR{L}_1^{*}}d{\mu }_1(\mathrm{if}\mathrm{no}\mathrm{prior}\mathrm{information})\hfill \\ & \propto -{\int }_{D}AR{L}_1\left(T\right)d{\mu }_1.\hfill \end{array}$$

We choose a number of possible values of ${\mu }_1$ as a representation in the numerical simulations and compute the out-of-control $AR{L}_1$. In this view, we can also conclude that the performance of the control charts depends on the value of the out-of-control average run length ($AR{L}_1$); the smaller the value, the better the performance.

Remark 1.

In this section, we compare the performance with different control charts, and the performance is measured by the  $ARL$. It is well known that the bandwidth has an impact on the kernel function estimation. The optimal bandwidth h in regression problems has been studied in [29,30]. However, in the monitoring problem discussed in this paper, the stopping rule is that if an abrupt change occurs, we issue a warning whenever the statistic exceeds the control limit. In the case of the same $AR{L}_0$, the control limit was also influenced by h, so if an abrupt change occurs, the bandwidth h in Gaussian and Laplace kernel functions has a joint effect on the statistic and the control limit, and their effects theoretically cancel each other out.

In this section, we compare the Gaussian kernel-based CUSUM multi-chart, Laplace kernel-based CUSUM multi-chart, and other existing control charts. The performance (i.e., detectability) is measured by the $ARL$. Since the out-of-control state is unknown before change detection, we choose several representative out-of-control states in the simulation.

3.1. The Case for Multivariate Normal Distribution

In this section, we present simulation results to demonstrate the performance of the Gaussian kernel-based CUSUM multi-chart (KMulti-Gaussian), the Laplace kernel-based CUSUM multi-chart (KMulti-Laplace), the distribution-free multivariate goodness-of-fit chart (DFMGoF), the self-starting EWMAC (SSEWMAC) chart [31], the chart based on the change-point and generalized likelihood ratio test (short for ChangePt [32]), and the RTC chart [16]. In the simulation, we consider the Gaussian and Laplace kernel functions-based CUSUM multi-charts with DFMGoF, RTC, ChangePt, and SSEWMAC. We compare the values of $AR{L}_1$ under the case that $AR{L}_0\approx 200$ and $m=100$. The bandwidth in the Laplace and Gaussian kernels is set to $h=1$.

Without loss of generality, the mean vector ${\mu }_0$ is set to $\mathbf{0}$, and the covariance matrix ${\Sigma }_0=I$ is chosen to be diagonal. We consider $p=10,30$, representing the low-dimensional and high-dimensional cases, respectively. We compare the out-of-control performance of the six competing charts.

Similar to other MSPC studies, it is impossible to enumerate all the change patterns to allow a full-scale study of the charts’ performance. Following similar studies in the literature [2,20,31,32], we consider scenarios as examples: shifts in the process mean vector in the first $[p/5]$ components of size $\delta$, i.e., ${\mu }_1={\mu }_0+\delta e$ with $e={(1,\cdots ,1,0,\cdots ,0)}^T$. We compare their performance in detecting mean shifts of magnitude $\delta =0.25,0.5,1,2,4$, respectively.

The upper part of

Table 1. The comparison of the $AR{L}_1$ in detecting location shifts when $m=100$ for a multivariate normal distribution.

p	$\mathit{\delta }$	KMulti-Gaussian	KMulti-Laplace	DFMGoF	RTC	ChangePt	SSEWMAC
10	0.25	42.63	45.24	93.9	132	186	122
	0.5	15.77	16.97	36.1	51.8	134	70.3
	1	5.32	5.67	12.4	12.1	36.4	11.6
	2	1.84	1.85	5.83	6.23	12.4	4.04
	4	1.01	1.01	3.93	4.72	4.81	1.59
30	0.25	23.22	25.92	83.1	100	186	87.8
	0.5	7.71	8.33	20.6	24.7	148	22.9
	1	2.46	2.36	8.4	7.62	52.9	9.09
	2	1.03	1.03	4.08	4.82	23.9	4.6
	4	1.01	1.01	2.74	3.9	12.2	2.37

represents the value of the $AR{L}_1$ for different control charts for $p=10$ with the same value of $AR{L}_0\approx 200$. As we have elaborated before, the smaller the value of $AR{L}_1$, the better the detection performance, given the same $AR{L}_0$. Thus, from Table 1, we can conclude that KMulti-Gaussian outperforms the other five charts when $p=10$. The lower part of Table 1 represents the value of the $AR{L}_1$ for different control charts for $p=30$ with the same value of $AR{L}_0\approx 200$; the conclusion is entirely similar.

In our simulation setting, we observed that the charts generally have a smaller $AR{L}_1$ when $p=30$ than that when $p=10$ given the same $\delta$. This is because the detectability is largely determined by the Mahalanobis distance of the shifted mean vector from the IC mean vector, $\Delta ={({\mu }_1-{\mu }_0)}^T{\Sigma }_0^{-1}({\mu }_1-{\mu }_0)$ (see [31]). Given ${\Sigma }_0$ and the change pattern in our study, we have ${\Delta }_{p=30}>{\Delta }_{p=10}$ given the same $\delta$. This can partially explain the better performance when $p=30$.

Our proposed method has a priori information compared to other methods and extracts the information in the form of kernel functions, such that the simulation results are not unexpected.

3.2. The Case for Multivariate t Distribution

In this section, we consider the multivariate t distribution. We use $t_{p,ϵ}$, which represents multivariate t distribution with $ϵ$ degrees of freedom. Let pre-change observations belong to multivariate t distribution with 5 degrees of freedom, denoted by $t_{p,5}$. We also consider the scenario as follows: shifts in the process mean vector in the first $[p/5]$ components of size $\delta$, i.e., ${\mu }_1={\mu }_0+\delta e$ with $e={(1,\cdots ,1,0,\cdots ,0)}^T$. We compare their performance in detecting mean shifts of magnitude $\delta =0.25,0.5,1,2,4$, respectively.

Table 2. The comparison of the $AR{L}_1$ in detecting location shifts when $m_0=100$ for a multivariate t distribution.

p	$\mathit{\delta }$	KMulti-Gaussian	KMulti-Laplace	DFMGoF	RTC	ChangePt	SSEWMAC
10	0.25	60.75	51.60	140	151.6	188	162
	0.5	25.54	20.75	58.5	68.3	155	88.9
	1	9.04	7.40	16.4	15.1	59.9	21.4
	2	3.58	2.89	7.37	6.74	19.9	8.1
	4	1.45	2.01	4.42	5.21	8.21	4.11
30	0.25	63.83	43.05	127	136	184	140
	0.5	18.81	11.89	37.7	45.6	154	55.3
	1	6.70	4.14	11.5	9.38	64.4	14.2
	2	2.76	1.55	5.46	5.77	31.5	6.43
	4	1.01	1.01	3.31	4.58	15.8	3.6

also compares our proposed methods with other existing methods.

The upper part of Table 2 represents the value of the $AR{L}_1$ for different control charts for $p=10$ with the same value of $AR{L}_0\approx 200$, and the lower part of Table 2 represents the value of the $AR{L}_1$ for different control charts for $p=30$ with the same value of $AR{L}_0\approx 200$. As we have elaborated before, the smaller the value of the $AR{L}_1$, the better the detection performance, given the same $AR{L}_0$. Thus, from Table 2, we can conclude that KMulti-Laplace outperforms the other five charts when $p=10$ and $p=30$.

3.3. The Case for Multivariate Gamma Distribution

In this section, we consider the multivariate Gamma distribution. We use $Gamm{a}_{p,ϵ}$, which represents the multivariate Gamma distribution with the shape parameter $ϵ$ and scale parameter of one. Let the pre-change observations belong to multivariate Gamma with the shape parameter of three and scale parameter of one, denoted by $Gamm{a}_{p,3}$. We also consider the scenario as follows: shifts in the process mean vector in the first $[p/5]$ components of size $\delta$, i.e., ${\mu }_1={\mu }_0+\delta e$ with $e={(1,\cdots ,1,0,\cdots ,0)}^T$. We compare their performance in detecting mean shifts of magnitude $\delta =0.25,0.5,1,2,4$, respectively.

The upper part of

Table 3. The comparison of the $AR{L}_1$ in detecting location shifts when $m_0=100$ for a multivariate Gamma distribution.

p	$\mathit{\delta }$	KMulti-Gaussian	KMulti-Laplace	DFMGoF	RTC	ChangePt	SSEWMAC
10	0.25	72.73	73.94	88.3	147	188	137
	0.5	35.76	35.03	26.8	59.3	152	67.2
	1	14.41	14.33	13.2	10.5	52.9	17.4
	2	5.69	5.62	7.27	6.43	17.2	7.39
	4	2.30	2.57	4.57	5.05	6.87	3.4
30	0.25	57.78	64.62	43.4	126	183	109
	0.5	19.84	24.56	17.2	28.4	159	35.7
	1	7.26	7.04	9.2	7.43	62.5	11.9
	2	2.62	2.35	5.16	5.25	27.8	5.72
	4	1.14	1.09	3.3	4.13	14.7	2.85

represents the value of the $AR{L}_1$ for different control charts for $p=10$ with the same value of $AR{L}_0\approx 200$, and the lower part of Table 3 represents the value of the $AR{L}_1$ for different control charts for $p=30$ with the same value of $AR{L}_0\approx 200$. As we have elaborated before, the smaller the value of the $AR{L}_1$, the better the detection performance, given the same $AR{L}_0$.

From Table 3, we can see that DFMGoF works better in some specific situations ($p=30$ and $\delta =0.25,0.5$), but note that from all the drift cases combined, KMulti-Gaussian is still better. We assume that all drift cases are equally likely. For example, the $AR{L}_1$ of DFMGoF is 14.38 less than the $AR{L}_1$ of KMulti-Gaussian when $\delta =0.25$ and $p=30$, but we set a weight of 0.25/4 when measuring performance in aggregate. Combining all the out-of-control states considered, KMulti-Gaussian and KMulti-Laplace are better than the other control charts.

From the three tables of the simulation results, we can conclude that the performance of the kernel-based multivariate nonparametric CUSUM multi-chart (KMulti-Gaussian and KMulti-Laplace) is better than the other four existing control charts, as the $AR{L}_1$ is smaller when the $AR{L}_0$ is the same.

Remark 2.

In this section, we compared the performance of different multivariate control charts. The performance of the methods proposed in this paper may not always be the best because the $KCPI$ in (5) is an integrated form for all possible post-change mean vectors ${\mu }_1$. Combining all the out-of-control states considered, the value of the $AR{L}_1$ for the method proposed in this paper is better compared with other existing control charts.

Remark 3.

The average run length (ARL) is a very important measurement in change detection. The in-control average run length, denoted by $AR{L}_0$, is, in the case of the observations, a pre-change observation. The out-of-control average run length, denoted by $AR{L}_1$, is, in the case of the observations, a post-change observation. The purpose of change detection is to alarm as quickly as possible if an abrupt change occurs. Hence, the smaller the value of the $AR{L}_1$, the better the performance of detection when the $AR{L}_0$ is the same.

4. Concluding Remarks

4.1. The Novelty of the Kernel-Based Multivariate CUSUM Multi-Chart

Traditional CUSUM multi-chart [19,28] is designed for one-dimensional data with a known distribution. Generalizing to multivariate data is not as straightforward. On the one hand, it requires some computation. On the other hand, we would like to design a multivariate CUSUM multi-chart that does not depend on the distribution. This paper proposes a kernel-based multivariate nonparametric CUSUM multi-chart to solve the following three challenges:

• In reality, we usually lack post-change information in online detection.
• Effectively capturing important features is essential when the dimension is $p=30$, a slightly larger number. The statistics, such as Hotelling’s $T^2$, may obtain the wrong alarm in online detection.
• In some cases, the amount of historical pre-change observations is not large.

4.2. Future Research Perspectives

We initially attempted to construct a control chart using the kernel function, which in turn was found to have good theoretical properties and simulation performance. We defined the kernel function-based CUSUM multi-chart and $KCPI$, which further gives the theoretical properties. In the simulation, we compared our proposed method with other existing methods and showed the advantages of our proposed method in terms of the average run length, a crucial index.

At the same time, it has some shortcomings. We assumed that multivariate data have prior knowledge. As the number of dimensions increases, acquiring prior knowledge becomes more and more difficult. For example, in the actual production process, there are many steps involved in production. We can monitor the sensor data from these steps as multivariate data, but acquiring more precise prior knowledge may necessitate a substantial investment of time and money.

We can conduct further research on the correlation between multivariate data internally and investigate the relationship between multidimensional variables when constructing control charts. This is a valuable research area to consider in the future. We can also construct parallel control charts using different kernel functions to monitor changes. In the simulation results above, the Gaussian kernel function and Laplace kernel function each have their own advantages and disadvantages. As to which kernel function works better, it depends on the data and how they change. In some data, this kernel function works well. In other data, another kernel function works well. The subsequent research intends to combine different kernel functions to construct parallel control charts.