Kernel Risk-Sensitive Mean p-Power Error Algorithms for Robust Learning

Abstract

As a nonlinear similarity measure defined in the reproducing kernel Hilbert space (RKHS), the correntropic loss (C-Loss) has been widely applied in robust learning and signal processing. However, the highly non-convex nature of C-Loss results in performance degradation. To address this issue, a convex kernel risk-sensitive loss (KRL) is proposed to measure the similarity in RKHS, which is the risk-sensitive loss defined as the expectation of an exponential function of the squared estimation error. In this paper, a novel nonlinear similarity measure, namely kernel risk-sensitive mean p-power error (KRP), is proposed by combining the mean p-power error into the KRL, which is a generalization of the KRL measure. The KRP with $p=2$ reduces to the KRL, and can outperform the KRL when an appropriate p is configured in robust learning. Some properties of KRP are presented for discussion. To improve the robustness of the kernel recursive least squares algorithm (KRLS) and reduce its network size, two robust recursive kernel adaptive filters, namely recursive minimum kernel risk-sensitive mean p-power error algorithm (RMKRP) and its quantized RMKRP (QRMKRP), are proposed in the RKHS under the minimum kernel risk-sensitive mean p-power error (MKRP) criterion, respectively. Monte Carlo simulations are conducted to confirm the superiorities of the proposed RMKRP and its quantized version.

1. Introduction

Online kernel-based learning is to extend the kernel methods to online settings where the data arrives sequentially, which has been widely applied in signal processing thanks to its excellent performance in addressing nonlinear issues [1]. The development of kernel methods is of great significance for practical applications. In kernel methods, the input data are transformed from the original space into the reproducing kernel Hilbert space (RKHS) using the kernel trick [2]. As the representative of the kernel methods, kernel adaptive filters (KAFs) provide an effective way to transform a nonlinear problem into a linear one, which have been widely introduced in system identification and time-series prediction [3,4,5]. Generally, KAFs are designed for Gaussian and non-Gaussian noises from the aspect of cost function, respectively.

For Gaussian noises, the second-order similarity measures of errors are generally used as a cost function of KAFs to achieve desirable filtering accuracy. Therefore, in the Gaussian noise environment, KAFs based on the second-order similarity measures of errors are mainly divided into three categories, i.e., the kernel least mean square (KLMS) algorithm [6], the kernel affine projection algorithm (KAPA) [7], and the kernel recursive least square algorithm (KRLS) [8]. However, the network size of KAFs increases linearly with the length of training, leading to large computational and storage burdens. To curb this structure growth, many sparsification methods are required, such as the surprise criterion (SC) [9], novelty criterion (NC) [10], coherence criterion [11], and approximate linear dependency (ALD) criterion [8]. However, these sparsification methods only discard the redundant data, leading to reduction of filtering accuracy. Unlike the aforementioned sparsification methods, the vector quantization (VQ) utilizes the redundant data to update the weights for accuracy improvement. The VQ is combined into KAFs to generate quantized KAFs, e.g., the quantized kernel least mean square algorithm (QKLMS) [12] and quantized kernel recursive least squares algorithm (QKRLS) [13].

However, the second-order similarity measures used in the aforementioned algorithms merely contain the second order statistics of errors, which cannot address non-Gaussian noises or outliers, efficiently [14]. Thus, it is very important to design a cost function beyond the second-order statistics of errors for combating non-Gaussian noises. The non-second order similarity measures can be divided into three categories, i.e., the mean p-power error (MPE) criterion [15], information theoretic learning (ITL) [14], and risk-sensitive loss (RL) based criteria [16,17]. The MPE criterion based on the pth absolute moment of the error can deal with non-Gaussian data with a proper p-value, efficiently. In general, MPE is robust to large outliers when $p<2$ [15], generating robust adaptive filters [15], e.g., the kernel least mean p-power (KLMP) algorithm [18] and the kernel recursive least mean p-power (KRLP) algorithm [18]. ITL can incorporate the complete distribution of errors into the learning process, resulting in the improvement of filtering precision and robustness to outliers. The most widely used ITL criterion is the maximum correntropy criterion (MCC) [19,20,21,22,23,24]. As a local similarity measure defined as a generalized correlation in the RKHS, the correntropy used in MCC can leverage higher order statistics of data to combat outliers [25]. However, the performance surface of the correntropic loss (C-Loss) is highly non-convex, which may lead to poor convergence performance. In the RL-based criteria, e.g., minimum risk-sensitive loss [16] and minimum kernel risk-sensitive loss (MKRL) [17,26], the risk-sensitive loss in the RKHS is convex extremely, which is more efficient for combating non-Gaussian noises or outliers than MCC [17,26]. However, since the MKRL uses the stochastic gradient descent (SGD) method to update its weights, the desirable filtering performance cannot be achieved for some complex nonlinear issues. The recursive update rules with excellent tracking ability can improve the filtering performance of adaptive filtering algorithms [21]. For example, KRLS based on the recursive update rule can improve the filtering performance of KLMS based on the SGD, significantly. To the best of our knowledge, however, it has not yet been proposed to design a recursive MKRL algorithm for desirable filtering performance in the RKHS by a recursive update rule.

In this paper, to inherit the advantages of both KRL and MPE for robustness improvement, we propose the risk-sensitive mean p-power error (RP) defined as the expectation of an exponential function of the pth absolute moment of the estimation error, and its kernel RP (KRP). The KRP can outperform the KRL by setting an appropriate p-value for robust learning, and the KRP with $p=2$ reduces to the KRL. The proposed KRP criterion is used to derive a novel recursive minimum kernel risk-sensitive mean p-power error (RMKRP) algorithm for desirable filtering performance by combining the weighted output information. Furthermore, to curb the growth of network size in the RMKRP, the VQ is combined into RMKRP to generate quantized RMKRP (QRMKRP).

The rest of this paper is organized as follows. In Section 2, we define the KRP, and give some basic properties. The KRP criterion is derived to develop a recursive adaptive algorithm by combining the weighted output information, called RMKRP algorithm in Section 3. To further reduce the network size of RMKRP, the vector quantization method is applied in RMKRP, thus generating the quantized RMKRP (QRMKRP) in Section 3. In Section 4, Monte Carlo simulations are conducted to validate the superiorities of the proposed algorithms in nonlinear examples. The conclusion is summarized in Section 5.

2. Kernel Risk-Sensitive Mean p-Power Error

2.1. Definition

According to [17], the risk-sensitive loss is defined in RKHS, called the kernel risk-sensitive loss (KRL). Given two arbitrary scalar random variables X and Y, where $X,Y\in \mathbb{R}$, the KRL is defined by

$$\begin{array}{cc}\hfill L_{\lambda }(X,Y)& =\frac{1}{\lambda }\mathbf{E}\left[exp\left(\lambda \left(\frac{1}{2}{∥\phi \left(X\right)-\phi \left(Y\right)∥}_{\mathbb{F}}^2\right)\right)\right]\hfill \\ & =\frac{1}{\lambda }\mathbf{E}\left[exp\left(\lambda \left(\frac{1}{2}{⟨\phi \left(X\right)-\phi \left(Y\right),\phi \left(X\right)-\phi \left(Y\right)⟩}_{\mathbb{F}}\right)\right)\right]\hfill \\ & =\frac{1}{\lambda }\mathbf{E}\left[exp\left(\lambda \left(\frac{1}{2}\left({⟨\phi \left(X\right),\phi \left(X\right)⟩}_{\mathbb{F}}+{⟨\phi \left(Y\right),\phi \left(Y\right)⟩}_{\mathbb{F}}-2{⟨\phi \left(X\right),\phi \left(Y\right)⟩}_{\mathbb{F}}\right)\right)\right)\right]\hfill \\ & =\frac{1}{\lambda }\mathbf{E}[exp\left(\lambda (1-{\kappa }_{\sigma }(X-Y))\right)]\hfill \\ & =\frac{1}{\lambda }\int exp\left(\lambda (1-{\kappa }_{\sigma }(X-Y))\right)d{F}_{XY}(x,y),\hfill \end{array}$$

(1)

where $\lambda >0$ is a risk-sensitive scalar parameter; $\phi \left(X\right)=\kappa (X,.)$ is a nonlinear mapping induced by a Mercer kernel ${\kappa }_{\sigma }(.)$, which transforms the data from the original space into the RKHS $\mathbb{F}$ equipped with an inner product ${⟨.,.⟩}_{\mathbb{F}}$ satisfying ${⟨\phi \left(X\right),\phi \left(Y\right)⟩}_{\mathbb{F}}={\phi }^{\mathrm{T}}\left(X\right)\phi \left(Y\right)={\kappa }_{\sigma }(X-Y)$; $\mathbf{E}$ denotes the mathematical expectation; ${∥.∥}_{\mathbb{F}}$ denotes the norm in RKHS $\mathbb{F}$; $F_{XY}(x,y)$ denotes the joint distribution function of $(X,Y)$. A shift-invariant Gaussian kernel ${\kappa }_{\sigma }(.)$ with bandwidth $\sigma$ is given as follows:

$$\begin{array}{c}\hfill {\kappa }_{\sigma }(x,y)={\kappa }_{\sigma }(x-y)=exp\left(-\frac{{(x-y)}^2}{2{\sigma }^2}\right).\end{array}$$

(2)

However, the joint distribution of $(X,Y)$ is usually unknown, and only N samples ${\{x\left(i\right),y\left(i\right)\}}_{i=1}^N$ are available. Hence, the nonparametric estimate of $L_{\lambda }(X,Y)$ is obtained by applying the Parzen windows [19] as ${\widehat{L}}_{\lambda }(X,Y)=\frac{1}{N\lambda }{\displaystyle \sum _{i=1}^N}exp\left(\lambda (1-{\kappa }_{\sigma }(x\left(i\right)-y\left(i\right)))\right)$. Note that the inner product in the RKHS for the same input is calculated by using kernel trick and (2), i.e, ${\phi }^{\mathrm{T}}\left(X\right)\phi \left(X\right)=exp\left(-\frac{{(X-X)}^2}{2{\sigma }^2}\right)=1$.

In this paper, we define a new non-second order similarity measure in the RKHS, i.e., the kernel risk-sensitive mean p-power error (KRP) loss. Given two random variables X and Y, the KRP loss is defined by

$$\begin{array}{cc}\hfill L_{\lambda ,p}(X,Y)& =\frac{1}{\lambda }\mathbf{E}\left[exp\left(\lambda 2^{-p/2}\left({∥\phi \left(X\right)-\phi \left(Y\right)∥}_{\mathbb{F}}^p\right)\right)\right]\hfill \\ & =\frac{1}{\lambda }\mathbf{E}\left[exp\left(\lambda 2^{-p/2}{\left({∥\phi \left(X\right)-\phi \left(Y\right)∥}_{\mathbb{F}}^2\right)}^{p/2}\right)\right]\hfill \\ & =\frac{1}{\lambda }\mathbf{E}[exp\left(\lambda 2^{-p/2}{(2-2{\kappa }_{\sigma }(X-Y))}^{p/2}\right)]\hfill \\ & =\frac{1}{\lambda }\mathbf{E}[exp\left(\lambda {(1-{\kappa }_{\sigma }(X-Y))}^{p/2}\right)]\hfill \\ & =\frac{1}{\lambda }\int exp\left(\lambda {(1-{\kappa }_{\sigma }(X-Y))}^{p/2}\right)d{F}_{X,Y}(x,y),\hfill \end{array}$$

(3)

where $p>0$ is the power parameter. Note that the KRL can be regarded as a special case of the KRP with $p=2$.

However, the joint distribution of X and Y is usually unknown in practice. Hence, the empirical KRP is defined as follows:

$$\begin{array}{c}\hfill {\widehat{L}}_{\lambda ,p}(X,Y)=\frac{1}{N\lambda }{\displaystyle \sum _{i=1}^N}exp\left(\lambda {(1-{\kappa }_{\sigma }(x\left(i\right)-y\left(i\right)))}^{p/2}\right),\end{array}$$

(4)

where ${\{x\left(i\right),y\left(i\right)\}}_{i=1}^N$ denotes the available finite number of samples. The empirical KRP can be regarded as a distance between both $\mathbf{X}={[x\left(1\right),x\left(2\right),\dots ,x\left(N\right)]}^{\mathrm{T}}$ and $\mathbf{Y}={[y\left(1\right),y\left(2\right),\dots ,y\left(N\right)]}^{\mathrm{T}}$.

2.2. Properties

In the following, we give some important properties of the proposed KRP.

Property 1.

$L_{\lambda ,p}(X,Y)$ is symmetric that is $L_{\lambda ,p}(X,Y)=L_{\lambda ,p}(Y,X)$.

Proof. 

Straightforward since ${\kappa }_{\sigma }(X-Y)={\kappa }_{\sigma }(Y-X)$. □

Property 2.

$L_{\lambda ,p}(X,Y)$ is positive and bounded, i.e., $\frac{1}{\lambda }\le L_{\lambda ,p}(X,Y)\le \frac{1}{\lambda }exp\left(\lambda \right)$, and reaches its minimum if $X=Y$.

Proof. 

Straightforward since $0<{\kappa }_{\sigma }(X-Y)\le 1$, and ${\kappa }_{\sigma }(X-Y)=1$ if $X=Y$. □

Property 3.

As λ is small enough, it holds that $L_{\lambda ,p}(X,Y)\approx \frac{1}{\lambda }+\mathbf{E}\left[{(1-{\kappa }_{\sigma }(X-Y))}^{p/2}\right]$.

Proof. 

For a small enough $\lambda$, we have $\lambda {(1-{\kappa }_{\sigma }(X-Y))}^{p/2}\to 0$, i.e.,

$$\begin{array}{c}\hfill exp(\lambda {(1-{\kappa }_{\sigma }(X-Y))}^{p/2})\approx 1+\lambda {(1-{\kappa }_{\sigma }(X-Y))}^{p/2}.\end{array}$$

(5)

Therefore, we can obtain

$$\begin{array}{cc}\hfill L_{\lambda ,p}(X,Y)& =\frac{1}{\lambda }\mathrm{E}[exp(\lambda {(1-{\kappa }_{\sigma }(X-Y))}^{p/2})]\hfill \\ & \stackrel{\left(5\right)}{\approx }\frac{1}{\lambda }\mathrm{E}[1+\lambda {(1-{\kappa }_{\sigma }(X-Y))}^{p/2}]\hfill \\ & =\frac{1}{\lambda }+\mathrm{E}\left[{(1-{\kappa }_{\sigma }(X-Y))}^{p/2}\right].\hfill \end{array}$$

(6)

 □

Property 4.

As σ is large enough, it holds that $L_{\lambda ,p}(X,Y)\approx \frac{1}{\lambda }+{\left(2{\sigma }^2\right)}^{-p/2}\mathbf{E}\left[\right|X-Y{|}^p]$.

Proof. 

Since $exp\left(x\right)$ is approximated by $1+x$ for a small enough x, for the case of large enough $\sigma$, i.e., $\frac{{(X-Y)}^2}{2{\sigma }^2}\to 0$. Thus, we can obtain the approximation as

$$\begin{array}{c}\hfill exp\left(-\frac{{(X-Y)}^2}{2{\sigma }^2}\right)\approx 1-\frac{{(X-Y)}^2}{2{\sigma }^2}.\end{array}$$

(7)

Similarly, when $\lambda {\left(\frac{{(X-Y)}^2}{2{\sigma }^2}\right)}^{p/2}\to 0$ for large enough $\sigma$, we can also obtain the approximation as

$$\begin{array}{c}\hfill exp\left(\lambda {\left(\frac{{(X-Y)}^2}{2{\sigma }^2}\right)}^{p/2}\right)\approx 1+\lambda {\left(\frac{{(X-Y)}^2}{2{\sigma }^2}\right)}^{p/2}.\end{array}$$

(8)

According to (7) and (8), we have

$$\begin{array}{cc}\hfill L_{\lambda ,p}(X,Y)& =\frac{1}{\lambda }\mathrm{E}\left[exp\left(\lambda {\left(1-exp\left(-\frac{{(X-Y)}^2}{2{\sigma }^2}\right)\right)}^{p/2}\right)\right]\hfill \\ & \stackrel{\left(7\right)}{\approx }\frac{1}{\lambda }\mathrm{E}\left[exp\left(\lambda {\left(\frac{{(X-Y)}^2}{2{\sigma }^2}\right)}^{p/2}\right)\right]\hfill \\ & \stackrel{\left(8\right)}{\approx }\frac{1}{\lambda }\mathrm{E}\left[1+\lambda {\left(\frac{{(X-Y)}^2}{2{\sigma }^2}\right)}^{p/2}\right]\hfill \\ & =\frac{1}{\lambda }+{\left(2{\sigma }^2\right)}^{-p/2}\mathrm{E}\left[{\left|X-Y\right|}^p\right].\hfill \end{array}$$

(9)

 □

Remark 1.

According to Properties 3 and 4, the KRP is, approximately, equivalent to the KMPE [27] as λ is small enough, and equivalent to the MPE [15] as σ is large enough. Thus, the KMPE and MPE can be viewed as two extreme cases of the KRP.

Property 5.

As p is small enough, it holds that $L_{\lambda ,p}(X,Y)\approx \frac{1}{\lambda }exp\left(\lambda (1+\left(p/2)\mathbf{E}\left[\log \right(1-{\kappa }_{\sigma }(X-Y)\right)]\right))\approx \frac{1}{\lambda }exp\left(\lambda \right)$.

Proof. 

Property 5 holds because of $(1-{\kappa }_{\sigma }{(X-Y))}^{p/2}\approx 1+\left(p/2)\mathbf{E}\left[\log \right(1-{\kappa }_{\sigma }(X-Y)\right)]\approx 1$. □

Property 6.

Let $\mathbf{e}=\mathbf{X}-\mathbf{Y}={[e\left(1\right),e\left(2\right),\dots ,e\left(N\right)]}^{\mathrm{T}}$, where $e\left(i\right)=x\left(i\right)-y\left(i\right)$. The empirical KRP ${\widehat{L}}_{\lambda ,p}(\mathbf{X},\mathbf{Y})$ as a function of $\mathbf{e}$ is convex at any point satisfying ${∥\mathbf{e}∥}_{\infty }={max}_{i=1,2,\dots ,N}\left|e\left(i\right)\right|\le \sigma$ and $p\ge 2$. When ${∥\mathbf{e}∥}_{\infty }>\sigma$, the empirical KRP ${\widehat{L}}_{\lambda ,p}(\mathbf{X},\mathbf{Y})$ is also convex if the risk-sensitive parameter $\lambda >0$ and power parameter $p\ge \underset{i=1,2,\dots ,N}{max}\left\{\frac{2(e^2\left(i\right)-{\sigma }^2)(1-{\kappa }_{\sigma }\left(e\left(i\right)\right))}{e^2\left(i\right){\kappa }_{\sigma }\left(e\left(i\right)\right)}+2\right\}$.

Proof. 

Since ${\widehat{L}}_{\lambda ,p}(\mathbf{X},\mathbf{Y})=\frac{1}{N\lambda }{\displaystyle \sum _{i=1}^N}exp(\lambda {(1-{\kappa }_{\sigma }\left(e\left(i\right)\right))}^{p/2})$, the Hessian matrix of ${\widehat{L}}_{\lambda ,p}(\mathbf{X},\mathbf{Y})$ with respect to $\mathbf{e}$ can be derived as

$$\begin{array}{c}\hfill {\mathrm{H}}_{{\widehat{L}}_{\lambda ,p}(\mathbf{X},\mathbf{Y})}\left(\mathbf{e}\right)=\left[\frac{{\partial }^2{\widehat{L}}_{\lambda ,p}(\mathbf{X},\mathbf{Y})}{\partial e\left(i\right)\partial e\left(j\right)}\right]=\mathrm{diag}[{\gamma }_1,{\gamma }_2,\dots ,{\gamma }_N],\end{array}$$

(10)

where

$$\begin{array}{cc}\hfill {\gamma }_i=& {\zeta }_i\left(\frac{p\lambda }{2{\sigma }^2}{(1-{\kappa }_{\sigma }\left(e\left(i\right)\right))}^{(p-2)/2}exp\left(-\frac{e^2\left(i\right)}{2{\sigma }^2}\right)e^2\left(i\right)+1\right.\hfill \\ & +\left.\frac{p-2}{2}{(1-{\kappa }_{\sigma }\left(e\left(i\right)\right))}^{-1}exp\left(-\frac{e^2\left(i\right)}{2{\sigma }^2}\right)\frac{e^2\left(i\right)}{{\sigma }^2}-\frac{e^2\left(i\right)}{{\sigma }^2}\right),\hfill \end{array}$$

(11)

with ${\zeta }_i=\frac{p}{2N{\sigma }^2}exp(\lambda {(1-{\kappa }_{\sigma }\left(e\left(i\right)\right))}^{p/2}){(1-{\kappa }_{\sigma }\left(e\left(i\right)\right))}^{(p-2)/2}exp\left(-\frac{e^2\left(i\right)}{2{\sigma }^2}\right),i=1,2,\dots ,N$. When $p\ge 2$, we have ${\mathrm{H}}_{{\widehat{L}}_{\lambda ,p}(\mathbf{X},\mathbf{Y})}\left(\mathbf{e}\right)>0$ if $\underset{i=1,2,\dots ,N}{max}\left|e\left(i\right)\right|\le \sigma$. From (11), if $\left|e\right(i\left)\right|\le \sigma$ and $p\ge 2$, or $\left|e\right(i)|>\sigma$ and $p\ge \left[2(e^2\left(i\right)-{\sigma }^2)(1-{\kappa }_{\sigma }\left(e\left(i\right)\right))/(e^2\left(i\right){\kappa }_{\sigma }\left(e\left(i\right)\right)\right)]+2$, we have ${\zeta }_i\ge 0$. Therefore, we have ${\mathrm{H}}_{{\widehat{L}}_{\lambda ,p}(X,Y)}\left(\mathbf{e}\right)\ge 0$ if

$$\begin{array}{c}\hfill p(\lambda {(1-{\kappa }_{\sigma }\left(e\left(i\right)\right))}^{p/2}+1)\ge \underset{i=1,2,\dots ,N}{max}\left\{\frac{2(e^2\left(i\right)-{\sigma }^2)(1-{\kappa }_{\sigma }\left(e\left(i\right)\right))}{e^2\left(i\right){\kappa }_{\sigma }\left(e\left(i\right)\right)}+2\right\},\end{array}$$

(12)

where $(\lambda {(1-{\kappa }_{\sigma }\left(e\left(i\right)\right))}^{p/2}+1)\ge 1$. Thus, we have $p\ge \underset{i=1,2,\dots ,N}{max}\left\{\frac{2(e^2\left(i\right)-{\sigma }^2)(1-{\kappa }_{\sigma }\left(e\left(i\right)\right))}{e^2\left(i\right){\kappa }_{\sigma }\left(e\left(i\right)\right)}+2\right\}$. □

Remark 2.

According to Property 6, the empirical KRP as a function of $\mathbf{e}$ is convex at any point satisfying ${∥\mathbf{e}∥}_{\infty }\le \sigma$ and $p\ge 2$. For the case ${∥\mathbf{e}∥}_{\infty }>\sigma$, the empirical KRP can still be convex at a point if the risk-sensitive parameter $\lambda >0$ and power parameter $p\ge \underset{i=1,2,\dots ,N}{max}\left\{\frac{2(e^2\left(i\right)-{\sigma }^2)(1-{\kappa }_{\sigma }\left(e\left(i\right)\right))}{e^2\left(i\right){\kappa }_{\sigma }\left(e\left(i\right)\right)}+2\right\}$.

Property 7.

As $\sigma \to \infty$ or $x\left(i\right)\to 0$, $i=1,2,\dots ,N$, it holds that

$$\begin{array}{c}\hfill {\widehat{L}}_{\lambda ,p}(\mathbf{X},\mathbf{0})\approx \frac{1}{\lambda }+\frac{1}{N{\left(\sqrt{2}\sigma \right)}^p}{∥\mathbf{X}∥}_p^p,\end{array}$$

(13)

where $\mathbf{0}$ denotes an N-dimensional zero vector.

Proof. 

$$\begin{array}{cc}\hfill {\widehat{L}}_{\lambda ,p}(\mathbf{X},\mathbf{0})& =\frac{1}{N\lambda }{\displaystyle \sum _{i=1}^N}exp(\lambda {(1-{\kappa }_{\sigma }\left(x\left(i\right)\right))}^{p/2})\approx \frac{1}{N\lambda }{\displaystyle \sum _{i=1}^N}exp\left(\lambda {\left(1-\left(1-\frac{x^2\left(i\right)}{2{\sigma }^2}\right)\right)}^{p/2}\right)\hfill \\ & =\frac{1}{N\lambda }{\displaystyle \sum _{i=1}^N}exp\left(\lambda {\left(\frac{x^2\left(i\right)}{2{\sigma }^2}\right)}^{p/2}\right)\approx \frac{1}{N\lambda }{\displaystyle \sum _{i=1}^N}\left[1+\lambda {\left(\frac{x^2\left(i\right)}{2{\sigma }^2}\right)}^{p/2}\right]\hfill \\ & =\frac{1}{\lambda }+\frac{1}{N{\left(\sqrt{2}\sigma \right)}^p}{\displaystyle \sum _{i=1}^N}|x\left(i\right){|}^p=\frac{1}{\lambda }+\frac{1}{N{\left(\sqrt{2}\sigma \right)}^p}{∥\mathbf{X}∥}_p^p.\hfill \end{array}$$

(14)

 □

Remark 3.

According to Property 7, the empirical KRP ${\widehat{L}}_{\lambda ,p}(\mathbf{X},\mathbf{0})$ behaves like an $L_p$ norm of $\mathbf{X}$ when kernel bandwidth σ is large enough.

3. Application to Adaptive Filtering

In this section, to combat non-Gaussian noises, two recursive robust adaptive algorithms under the proposed KRP criterion are proposed in the RKHS using the kernel trick and vector quantization technique, respectively.

3.1. RMKRP

The recursive strategy is introduced into the KRP loss function, namely the recursive minimum kernel risk-sensitive mean p-power error (RMKRP) algorithm. The offline solution to minimum of the KRP loss is first obtained. Based on the obtained offline solution, the recursive solution or online solution to minimum of the KRP loss is then derived using some matrix operations, which generates the RMKRP algorithm. The details of RMKRP are shown as follows.

Consider the prediction of a continuous input-output model $f:\mathbb{U}\to \mathbb{R}$ based on adaptive filtering shown in Figure 1, where $\mathbf{u}\left(i\right)\in \mathbb{U}\subset {\mathbb{R}}^D$ is the ith D-dimensional input vector, $d\left(i\right)\in \mathbb{R}$ is the ith scalar desired output contaminated by a noise $v\left(i\right)$, i.e., $d\left(i\right)=f\left(\mathbf{u}\right(i\left)\right)+v\left(i\right)$. A sequence of training samples ${\left\{\mathbf{u}\left(j\right),d\left(j\right)\right\}}_{j=1}^i$ is used to perform the prediction of $f(·)$ in an adaptive filter. The nonlinear mapping $\mathit{\phi }\left(\mathbf{u}\right(j\left)\right)$ of input $\mathbf{u}\left(j\right)$ is denoted by $\mathit{\phi }\left(j\right)$ for simplicity. Hence, in the RKHS $\mathbb{F}$, the training samples are changed to $\left\{\mathsf{\Phi }\left(i\right),\mathbf{d}\left(i\right)\right\}$, where the desired output vector is $\mathbf{d}\left(i\right)={[d\left(1\right),d\left(2\right),\dots ,d\left(i\right)]}^{\mathrm{T}}$ and the input kernel mapping matrix is $\mathsf{\Phi }\left(i\right)=\left[\mathit{\phi }\right(1),\mathit{\phi }(2),\dots ,\mathit{\phi }(i\left)\right]$. The prediction denoted by $\widehat{f}(·)$ in the RKHS is therefore given as $\widehat{f}(·)={\mathit{\phi }}^{\mathrm{T}}(·)\mathsf{\Omega }$, where $\mathsf{\Omega }\in \mathbb{F}$ is the weight vector in a high dimensional feature space $\mathbb{F}$.

Figure 1. Block diagram of adaptive filtering.

An exponentially-weighted loss function is used here to put more emphasis on recent data and to de-emphasize data on the remote past [28]. When $\left\{\mathsf{\Phi }\left(i\right),\mathbf{d}\left(i\right)\right\}$ are available, the weight vector $\mathsf{\Omega }\left(i\right)$ is obtained as the offline solution to minimizing the following weighted cost function:

$$\begin{array}{c}\hfill J\left(\mathsf{\Omega }\left(i\right)\right)={\displaystyle \sum _{j=1}^i}{\rho }^{i-j}\frac{1}{\lambda }exp\left(\lambda {\left(z\left(j\right)\right)}^{\frac{p}{2}}\right)+\frac{1}{2}{\rho }^i\zeta {∥\mathsf{\Omega }\left(i\right)∥}^2,\end{array}$$

(15)

where $\rho$ denotes the forgetting factor in the interval $[0,1]$, $\zeta$ is the regularization factor, $z\left(j\right)=1-exp\left(-\frac{e^2\left(j\right)}{2{\sigma }^2}\right)$, and $e\left(j\right)=d\left(j\right)-{\mathit{\phi }}^{\mathrm{T}}\left(j\right)\mathsf{\Omega }\left(i\right)$ denotes the jth estimate error. The second term is a norm penalizing term, which is to guarantee the existence of the inverse of the input data autocorrelation matrix especially during the initial update stages. In addition, the regularization term is weighted by $\rho$, which deemphasizes regularization as time progresses. According to Property 6, the empirical KRP as a function of $\mathbf{e}$ is convex at any point satisfying $\underset{j=1,2,\dots ,i}{max}\left|e\left(j\right)\right|\le \sigma$, $\lambda >0$, and $p\ge 2$. To obtain the minimum of (15), its gradient is calculated, i.e.,

$$\begin{array}{cc}\hfill \frac{\partial J\left(\mathsf{\Omega }\right(i\left)\right)}{\partial \mathsf{\Omega }\left(i\right)}=& -\frac{p}{2{\sigma }^2}{\displaystyle \sum _{j=1}^i}\mathit{\phi }\left(j\right){\rho }^{i-j}exp\left(\lambda {\left(z\left(j\right)\right)}^{\frac{p}{2}}\right){\left(z\left(j\right)\right)}^{\frac{p-2}{2}}(d\left(j\right)-{\mathit{\phi }}^{\mathrm{T}}\left(j\right)\mathsf{\Omega }\left(i\right))(1-z\left(j\right))+{\rho }^i\zeta \mathsf{\Omega }\left(i\right)\hfill \\ \hfill =& -\frac{p}{2{\sigma }^2}{\displaystyle \sum _{j=1}^i}\mathit{\phi }\left(j\right){\rho }^{i-j}exp\left(\lambda {\left(z\left(j\right)\right)}^{\frac{p}{2}}\right){\left(z\left(j\right)\right)}^{\frac{p-2}{2}}(1-z\left(j\right))d\left(j\right)\hfill \\ & +\frac{p}{2{\sigma }^2}{\displaystyle \sum _{j=1}^i}\mathit{\phi }\left(j\right){\rho }^{i-j}exp\left(\lambda {\left(z\left(j\right)\right)}^{\frac{p}{2}}\right){\left(z\left(j\right)\right)}^{\frac{p-2}{2}}(1-z\left(j\right)){\mathit{\phi }}^{\mathrm{T}}\left(j\right)\mathsf{\Omega }\left(i\right)+{\rho }^i\zeta \mathsf{\Omega }\left(i\right).\hfill \end{array}$$

(16)

Setting (16) to zero, i.e., $\frac{\partial J\left(\mathsf{\Omega }\right)}{\partial \mathsf{\Omega }}=0$, we can obtain the offline solution to minimum of (15) as follows:

$$\begin{array}{c}\hfill \mathsf{\Omega }\left(i\right)={(\mathsf{\Phi }\left(i\right)\mathbf{H}\left(i\right){\mathsf{\Phi }}^{\mathrm{T}}\left(i\right)+{\rho }^i\zeta 2{\sigma }^2/p\mathbf{I})}^{-1}\mathsf{\Phi }\left(i\right)\mathbf{H}\left(i\right)\mathbf{d}\left(i\right),\end{array}$$

(17)

where $\mathbf{H}\left(i\right)=\mathrm{diag}[H_1\left(i\right),H_2\left(i\right),\dots ,H_i\left(i\right)]$ with $H_j\left(i\right)={\rho }^{i-j}exp\left(\lambda {\left(z\left(j\right)\right)}^{\frac{p}{2}}\right){\left(z\left(j\right)\right)}^{\frac{p-2}{2}}(1-z\left(j\right))$, $j=1,2,\dots ,i$, and $\mathbf{I}$ denotes an identity matrix with an appropriate dimension.

To obtain an efficient recursive solution to the minimum of (15), a Mercer kernel is used to construct the RKHS. Here, the Gaussian kernel is used as a Mercer kernel, which is denoted as ${\kappa }_{{\sigma }_1}(.)$ with ${\sigma }_1$ being the kernel width. The inner product in the RKHS can be calculated by using the kernel trick [28], i.e., ${\kappa }_{{\sigma }_1}(\mathbf{u}\left(i\right),\mathbf{u}\left(j\right))={\kappa }_{{\sigma }_1}(\mathbf{u}\left(i\right)-\mathbf{u}\left(j\right))={\phi }^{\mathrm{T}}\left(\mathbf{u}\left(i\right)\right)\phi \left(\mathbf{u}\left(j\right)\right)={\mathit{\phi }}^{\mathrm{T}}\left(i\right)\mathit{\phi }\left(j\right)$, efficiently, which can avoid the direct calculation of nonlinear mapping $\mathit{\phi }(·)$.

Since the matrix inversion lemma [28] is described by ${(\mathbf{A}+\mathbf{BCD})}^{-1}={\mathbf{A}}^{-1}-{\mathbf{A}}^{-1}\mathbf{B}{({\mathbf{C}}^{-1}+\mathbf{D}{\mathbf{A}}^{-1}\mathbf{B})}^{-1}\mathbf{D}{\mathbf{A}}^{-1}$, by letting $\mathbf{A}={\rho }^i\zeta 2{\sigma }^2/p\mathbf{I}$, $\mathbf{B}=\mathsf{\Phi }\left(i\right)$, $\mathbf{C}=\mathbf{H}\left(i\right)$, and $\mathbf{D}={\mathsf{\Phi }}^{\mathrm{T}}\left(i\right)$, we rewrite (17) as

$$\begin{array}{c}\hfill {(\mathsf{\Phi }\left(i\right)\mathbf{H}\left(i\right){\mathsf{\Phi }}^{\mathrm{T}}\left(i\right)+{\rho }^i\zeta 2{\sigma }^2/p\mathbf{I})}^{-1}\mathsf{\Phi }\left(i\right)\mathbf{H}\left(i\right)=\mathsf{\Phi }\left(i\right){({\mathsf{\Phi }}^{\mathrm{T}}\left(i\right)\mathsf{\Phi }\left(i\right)+{\rho }^i\zeta 2{\sigma }^2/p\mathbf{H}{\left(i\right)}^{-1})}^{-1}.\end{array}$$

(18)

Substituting (18) into (17) yields

$$\begin{array}{c}\hfill \mathsf{\Omega }\left(i\right)=\mathsf{\Phi }\left(i\right){({\mathsf{\Phi }}^{\mathrm{T}}\left(i\right)\mathsf{\Phi }\left(i\right)+{\rho }^i\zeta 2{\sigma }^2/p\mathbf{H}{\left(i\right)}^{-1})}^{-1}\mathbf{d}\left(i\right).\end{array}$$

(19)

Note that ${\mathsf{\Phi }}^{\mathrm{T}}\left(i\right)\mathsf{\Phi }\left(i\right)$ in (19) can be computed by the kernel trick, efficiently. The weight vector $\mathsf{\Omega }\left(i\right)$ is therefore described explicitly as a linear combination of the input data in the RKHS, i.e.,

$$\begin{array}{c}\hfill \mathsf{\Omega }\left(i\right)=\mathsf{\Phi }\left(i\right)\mathit{\alpha }\left(i\right),\end{array}$$

(20)

where $\mathit{\alpha }\left(i\right)$ denotes the coefficients vector.

It can be seen from (20) that the recursive form of $\mathsf{\Omega }\left(i\right)$ is changed to that of $\mathit{\alpha }\left(i\right)$. Hence, in the following, the key for finding a recursive solution to the minimum of (15) is to obtain the recursive form of $\mathit{\alpha }\left(i\right)$.

The coefficients vector $\mathit{\alpha }\left(i\right)$ is calculated using the kernel trick as

$$\begin{array}{c}\hfill \mathit{\alpha }\left(i\right)={({\mathsf{\Phi }}^{\mathrm{T}}\left(i\right)\mathsf{\Phi }\left(i\right)+{\rho }^i\zeta 2{\sigma }^2/p\mathbf{H}{\left(i\right)}^{-1})}^{-1}\mathbf{d}\left(i\right).\end{array}$$

(21)

For simplicity, we obtain the update form of $\mathit{\alpha }\left(i\right)$ indirectly by defining $\mathbf{\Lambda }\left(i\right)$ as

$$\begin{array}{c}\hfill \mathbf{\Lambda }\left(i\right)={({\mathsf{\Phi }}^{\mathrm{T}}\left(i\right)\mathsf{\Phi }\left(i\right)+{\rho }^i\zeta 2{\sigma }^2/p\mathbf{H}{\left(i\right)}^{-1})}^{-1},\end{array}$$

(22)

where $\mathsf{\Phi }\left(i\right)=\left\{\mathsf{\Phi }\right(i-1),\mathit{\phi }(i\left)\right\}$. Then, the update form of $\mathbf{\Lambda }\left(i\right)$ can be further obtained

$$\begin{array}{c}\hfill \mathbf{\Lambda }\left(i\right)=\left[\begin{array}{c}{\mathsf{\Phi }}^{\mathrm{T}}(i-1)\mathsf{\Phi }(i-1)+{\rho }^i\zeta 2{\sigma }^2/p\mathbf{H}{(i-1)}^{-1}\\ {\mathit{\phi }}^{\mathrm{T}}\left(i\right)\mathsf{\Phi }(i-1)\end{array}\right.{\left.\begin{array}{c}{\mathsf{\Phi }}^{\mathrm{T}}(i-1)\mathit{\phi }\left(i\right)\\ {\mathit{\phi }}^{\mathrm{T}}\left(i\right)\mathit{\phi }\left(i\right)+{\rho }^i\zeta 2{\sigma }^2/p\nu \left(i\right)\end{array}\right]}^{-1},\end{array}$$

(23)

where $\nu \left(i\right)={\left(exp\left(\lambda {\left(z\left(i\right)\right)}^{\frac{p}{2}}\right){\left(z\left(i\right)\right)}^{\frac{p-2}{2}}(1-z\left(i\right))\right)}^{-1}$. By using some matrix operations, we further simplify (23) as

$$\begin{array}{c}\hfill \mathbf{\Lambda }{\left(i\right)}^{-1}=\left[\begin{array}{c}\mathbf{\Lambda }{(i-1)}^{-1}\\ {\mathit{\xi }}^{\mathrm{T}}\left(i\right)\end{array}\right.\left.\begin{array}{c}\mathit{\xi }\left(i\right)\\ {\rho }^i\zeta 2{\sigma }^2/p\nu \left(i\right)+{\mathit{\phi }}^{\mathrm{T}}\left(i\right)\mathit{\phi }\left(i\right)\end{array}\right],\end{array}$$

(24)

where $\mathit{\xi }\left(i\right)={\mathsf{\Phi }}^{\mathrm{T}}(i-1)\mathit{\phi }\left(i\right)$. By using the following block matrix inversion identity [18,21,28]

$$\begin{array}{c}\hfill \left[\begin{array}{c}\mathbf{A}\\ \mathbf{C}\end{array}\right.{\left.\begin{array}{c}\mathbf{B}\\ \mathbf{D}\end{array}\right]}^{-1}=\left[\begin{array}{c}{(\mathbf{A}-\mathbf{B}{\mathbf{D}}^{-1}\mathbf{C})}^{-1}\\ -{\mathbf{D}}^{-1}\mathbf{C}{(\mathbf{A}-\mathbf{B}{\mathbf{D}}^{-1}\mathbf{C})}^{-1}\end{array}\right.\left.\begin{array}{c}-{\mathbf{A}}^{-1}\mathbf{B}{(\mathbf{D}-\mathbf{C}{\mathbf{A}}^{-1}\mathbf{B})}^{-1}\\ {(\mathbf{D}-\mathbf{C}{\mathbf{A}}^{-1}\mathbf{B})}^{-1}\end{array}\right],\end{array}$$

(25)

then, we can obtain the update equation for the inverse of the growing matrix in (24) as

$$\begin{array}{c}\hfill \mathbf{\Lambda }\left(i\right)=r^{-1}\left(i\right)\left[\begin{array}{c}\mathbf{\Lambda }(i-1)r\left(i\right)+\mathit{\theta }\left(i\right){\mathit{\theta }}^{\mathrm{T}}\left(i\right)\\ -{\mathit{\theta }}^{\mathrm{T}}\left(i\right)\end{array}\right.\left.\begin{array}{c}-\mathit{\theta }\left(i\right)\\ 1\end{array}\right],\end{array}$$

(26)

where $\mathit{\theta }\left(i\right)=\mathbf{\Lambda }(i-1)\mathit{\xi }\left(i\right)$ and $r\left(i\right)={\rho }^i\zeta 2{\sigma }^2/p\nu \left(i\right)+{\mathit{\phi }}^{\mathrm{T}}\left(i\right)\mathit{\phi }\left(i\right)-{\mathit{\theta }}^{\mathrm{T}}\left(i\right)\mathit{\xi }\left(i\right)$. Combining (21) with (26), the coefficients vector $\mathit{\alpha }\left(i\right)$ of the weight vector $\mathsf{\Omega }\left(i\right)$ is shown as follows:

$$\begin{array}{cc}\hfill \mathit{\alpha }\left(i\right)=\mathbf{\Lambda }\left(i\right)\mathbf{d}\left(i\right)& =\left[\begin{array}{c}\mathbf{\Lambda }(i-1)+\mathit{\theta }\left(i\right){\mathit{\theta }}^{\mathrm{T}}\left(i\right)r^{-1}\left(i\right)\\ -{\mathit{\theta }}^{\mathrm{T}}\left(i\right)r^{-1}\left(i\right)\end{array}\right.\left.\begin{array}{c}-\mathit{\theta }\left(i\right)r^{-1}\left(i\right)\\ r^{-1}\left(i\right)\end{array}\right]\left[\begin{array}{c}\mathbf{d}(i-1)\\ d\left(i\right)\end{array}\right]\hfill \\ & =\left[\begin{array}{c}\mathit{\alpha }(i-1)-\mathit{\theta }\left(i\right)r^{-1}\left(i\right)e\left(i\right)\\ r^{-1}\left(i\right)e\left(i\right)\end{array}\right],\hfill \end{array}$$

(27)

where $e\left(i\right)=d\left(i\right)-\widehat{f}\left(i\right)$ denotes the difference between the desired output $d\left(i\right)$ and the system output $\widehat{f}\left(i\right)=\mathit{\xi }{\left(i\right)}^{\mathrm{T}}\mathit{\alpha }(i-1)={\displaystyle \sum _{j=1}^{i-1}}{\mathit{\alpha }}_j(i-1){\kappa }_{{\sigma }_1}(\mathbf{u}\left(j\right),\mathbf{u}\left(i\right))$. ${\mathit{\alpha }}_j(i-1)$ is the jth element of $\mathit{\alpha }(i-1)$ and all the previous data are the centers. The coefficients $\mathit{\alpha }(i-1)$ and all the previous data should be stored at each iteration. Finally, the RMKRP algorithm is summarized in Algorithm 1.

Algorithm 1: The RMKRP Algorithm.

Initialization: $\left\{\mathbf{u}\right(i),d(i\left)\right\},i=1,2,\dots$ $p,\lambda ,\rho ,\sigma ,{\sigma }_1>0,\zeta \in [0,1],z\left(1\right)=1-exp\left(-\frac{d^2\left(1\right)}{2{\sigma }^2}\right)$. $H_1\left(1\right)=exp\left(\lambda {\left(z\left(1\right)\right)}^{\frac{p}{2}}\right){\left(z\left(1\right)\right)}^{\frac{p-2}{2}}(1-z\left(1\right))$. $\mathbf{\Lambda }\left(1\right)={(\zeta \rho 2{\sigma }^2/p/H_1\left(1\right)+\kappa (\mathbf{u}\left(1\right),\mathbf{u}\left(1\right)))}^{-1},\mathit{\alpha }\left(1\right)=\mathbf{\Lambda }\left(1\right)d\left(1\right)$. Computation: While {$\mathbf{u}\left(i\right),d\left(i\right)\left\}\right(i>1)$ available do   1) $\mathit{\xi }\left(i\right)={[\kappa (\mathbf{u}\left(1\right),\mathbf{u}\left(1\right)),...,\kappa (\mathbf{u}\left(i\right),\mathbf{u}(i-1))]}^{\mathrm{T}}$   2) $e\left(i\right)=d\left(i\right)-\mathit{\xi }{\left(i\right)}^{\mathrm{T}}\mathit{\alpha }(i-1)$   3) $\mathit{\theta }\left(i\right)=\mathbf{\Lambda }(i-1)\mathit{\xi }\left(i\right)$   4) $z\left(i\right)=1-exp\left(-\frac{e^2\left(i\right)}{2{\sigma }^2}\right)$   5) $\nu \left(i\right)={\left(exp\left(\lambda {\left(z\left(i\right)\right)}^{\frac{p}{2}}\right){\left(z\left(i\right)\right)}^{\frac{p-2}{2}}(1-z\left(i\right))\right)}^{-1}$   6) $r\left(i\right)={\rho }^i\zeta 2{\sigma }^2/p\nu \left(i\right)+\kappa (\mathbf{u}\left(i\right),\mathbf{u}\left(i\right))-{\mathit{\theta }}^{\mathrm{T}}\left(i\right)\mathit{\xi }\left(i\right)$   7) $\mathbf{\Lambda }\left(i\right)=r^{-1}\left(i\right)\left[\begin{array}{c}\mathbf{\Lambda }(i-1)r\left(i\right)+\mathit{\theta }\left(i\right){\mathit{\theta }}^{\mathrm{T}}\left(i\right)\\ -{\mathit{\theta }}^{\mathrm{T}}\left(i\right)\end{array}\right.\left.\begin{array}{c}-\mathit{\theta }\left(i\right)\\ 1\end{array}\right]$   8) $\mathit{\alpha }\left(i\right)=\left[\begin{array}{c}\mathit{\alpha }(i-1)-\mathit{\theta }\left(i\right)r^{-1}\left(i\right)e\left(i\right)\\ r^{-1}\left(i\right)e\left(i\right)\end{array}\right].$ end while

3.2. QRMKRP

The RMKRP algorithm generates a linearly growing network owing to the used kernel trick. The online vector quantization (VQ) method [12] has been successfully applied in KAFs to curb its network growth efficiently. Thus, we incorporate the online VQ method into the RMKRP to develop the quantized RMKRP (QRMKRP) algorithm, which is shown as follows.

Suppose that the dictionary $\mathbf{C}\left(i\right)$ contains L vectors at discrete time i, i.e., $\mathbf{C}\left(i\right)={\left\{{\mathbf{C}}_k\left(i\right)\right\}}_{k=1}^L$, $k\in Id=\{1,2,\dots ,L\}$, which means that there are L distinctive quantization regions. In the RKHS, the prediction $\widehat{f}\left(i\right)$ is therefore expressed as $\widehat{f}\left(i\right)={\phi }^{\mathrm{T}}\left({\mathbf{C}}_k\left(i\right)\right)\widehat{\mathsf{\Omega }}$, where $\widehat{\mathsf{\Omega }}\in \mathbb{F}$ is the weight vector in RKHS $\mathbb{F}$. The cost function of QRMKRP based on $\mathbf{C}\left(i\right)$ is denoted as

$$\begin{array}{c}\hfill {\displaystyle \sum _{k=1}^L}{\displaystyle \sum _{n=1}^{\left|{\mathrm{D}}_k\right|}}{\rho }^{i-k}\frac{1}{\lambda }exp\left(\lambda {\left(1-exp\left(-\frac{{(d_{kn}\left(i\right)-{\phi }^{\mathrm{T}}\left({\mathbf{C}}_k\left(i\right)\right)\widehat{\mathsf{\Omega }}\left(i\right))}^2}{2{\sigma }^2}\right)\right)}^{\frac{p}{2}}\right)+\frac{1}{2}{\rho }^i\zeta {∥\widehat{\mathsf{\Omega }}\left(i\right)∥}^2,\end{array}$$

(28)

where $\left|{\mathrm{D}}_k\right|$ denotes the number of input data those lie in the kth quantization region of $\mathbf{C}\left(i\right)$ and satisfies ${\sum }_{k\in Id}\left|{\mathrm{D}}_k\right|=i$ and $\left|{\mathrm{D}}_k\right|\ge 1$, and $d_{kn}\left(i\right)$ is the desired output $d\left(i\right)$ corresponding to the nth element within the kth quantization region.

The offline solution to the minimization of (28) can be described by

$$\begin{array}{c}\hfill \widehat{\mathsf{\Omega }}\left(i\right)={\left[\widehat{\mathsf{\Phi }}\left(i\right)\widehat{\mathbf{H}}\left(i\right){\widehat{\mathsf{\Phi }}}^{\mathrm{T}}\left(i\right)+{\rho }^i\zeta 2{\sigma }^2/p\mathbf{I}\right]}^{-1}\widehat{\mathsf{\Phi }}\left(i\right)\widehat{\mathbf{d}}\left(i\right),\end{array}$$

(29)

where $\widehat{\mathsf{\Phi }}\left(i\right)=[\phi \left({\mathbf{C}}_1\left(i\right)\right),\phi \left({\mathbf{C}}_2\left(i\right)\right),\dots ,\phi \left({\mathbf{C}}_L\left(i\right)\right)]$ with $L\ll i$ elements; $\widehat{\mathbf{H}}\left(i\right)=\mathrm{diag}[{\sum }_{n=1}^{|D_1|}{H}_{1n}\left(i\right),{\sum }_{n=1}^{|D_2|}{H}_{2n}\left(i\right),\dots ,{\sum }_{n=1}^{|D_L|}{H}_{Ln}\left(i\right)]$ denotes a accumulated diagonal matrix; $\widehat{\mathbf{d}}\left(i\right)=\mathrm{diag}{[{\sum }_{n=1}^{|D_1|}{H}_{1n}\left(i\right)d_{1n}\left(i\right),{\sum }_{n=1}^{|D_2|}{H}_{2n}\left(i\right)d_{2n}\left(i\right),\dots ,{\sum }_{n=1}^{|D_L|}{H}_{Ln}\left(i\right)d_{Ln}\left(i\right)]}^{\mathrm{T}}$ denotes a accumulated weighted output vector; $H_{kn}\left(i\right)$ denotes $H_i\left(i\right)$ corresponding to the nth entry of the kth quantization region; $\mathbf{I}$ denotes an identity matrix with an appropriate dimension. Since (29) has a similar form to (17), we simplify (29) as

$$\begin{array}{c}\hfill \widehat{\mathsf{\Omega }}\left(i\right)=\widehat{\mathsf{\Phi }}\left(i\right){\left[\widehat{\mathbf{H}}\left(i\right)\widehat{\mathbf{K}}\left(i\right)+{\rho }^i\zeta 2{\sigma }^2/p\mathbf{I}\right]}^{-1}\widehat{\mathbf{d}}\left(i\right)=\widehat{\mathsf{\Phi }}\left(i\right)\widehat{\mathbf{Q}}\left(i\right)\widehat{\mathbf{d}}\left(i\right)=\widehat{\mathsf{\Phi }}\left(i\right)\widehat{\mathit{\alpha }}\left(i\right),\end{array}$$

(30)

where $\widehat{\mathbf{K}}\left(i\right)={\widehat{\mathsf{\Phi }}}^{\mathrm{T}}\left(i\right)\widehat{\mathsf{\Phi }}\left(i\right)$. To obtain the recursive solution to the minimization of (28), we let $\widehat{\mathit{\alpha }}\left(i\right)=\widehat{\mathbf{Q}}\left(i\right)\widehat{\mathbf{d}}\left(i\right)$ and denote

$$\begin{array}{c}\hfill \widehat{\mathbf{Q}}\left(i\right)={\left[\widehat{\mathbf{H}}\left(i\right)\widehat{\mathbf{K}}\left(i\right)+{\rho }^i\zeta 2{\sigma }^2/p\mathbf{I}\right]}^{-1}.\end{array}$$

(31)

To update $\widehat{\mathsf{\Omega }}\left(i\right)$ in (30) recursively, two cases are therefore considered.

(1) First, Case: dis$\left(\mathbf{u}\right(i),\mathbf{C}(i-1\left)\right)\le ϵ$: In this case, we have $\mathbf{C}\left(i\right)=\mathbf{C}(i-1)$ and $\widehat{\mathbf{Q}}\left(i\right)=\widehat{\mathbf{Q}}(i-1)$, which means the input $\mathbf{u}\left(i\right)$ is therefore quantized to the $k^{*}$th element of dictionary $\mathbf{C}(i-1)$, where $k^{*}$ = $\underset{1\le k\le \left|{\mathbf{C}}_k\left(i-1\right)\right|}{argmin}{∥\mathbf{u}\left(i\right)-{\mathbf{C}}_k\left(i-1\right)∥}^2$. The matrix $\widehat{\mathbf{H}}\left(i\right)$ and the vector $\widehat{\mathbf{d}}\left(i\right)$ have a similar form to [13]. Here, $\widehat{\mathbf{H}}\left(i\right)$ and $\widehat{\mathbf{d}}\left(i\right)$ can be shown as

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\widehat{\mathbf{H}}\left(i\right)=\widehat{\mathbf{H}}(i-1)+H_i\left(i\right){\mathit{\tau }}_{k^{*}}{\mathit{\tau }}_{k^{*}}^{\mathrm{T}}\hfill \\ \widehat{\mathbf{d}}\left(i\right)=\widehat{\mathbf{d}}(i-1)+H_i\left(i\right)d\left(i\right){\mathit{\tau }}_{k^{*}}\hfill \end{array}\right.,\end{array}$$

(32)

where ${\mathit{\tau }}_{k^{*}}$ is a $\left|\mathbf{C}(i-1)\right|$-dimensional column vector whose $k^{*}$th element is 1 and all other elements are 0. Combining (32) into (31), the matrix $\widehat{\mathbf{Q}}\left(i\right)$ can be expressed as $\widehat{\mathbf{Q}}\left(i\right)={[\widehat{\mathbf{Q}}{(i-1)}^{-1}+H_i\left(i\right){\mathit{\tau }}_{k^{*}}{\mathit{\tau }}_{k^{*}}^{\mathrm{T}}\widehat{\mathbf{K}}(i-1)]}^{-1}$. By using the matrix inversion lemma [28], we obtain

$$\begin{array}{c}\hfill \widehat{\mathbf{Q}}\left(i\right)=\widehat{\mathbf{Q}}(i-1)-\frac{{\widehat{\mathbf{Q}}}_{k^{*}}(i-1){\widehat{\mathbf{K}}}_{k^{*}}^{\mathrm{T}}(i-1)\widehat{\mathbf{Q}}(i-1)}{H_i^{-1}\left(i\right)+{\widehat{\mathbf{K}}}_{k^{*}}^{\mathrm{T}}(i-1){\widehat{\mathbf{Q}}}_{k^{*}}(i-1)},\end{array}$$

(33)

where ${\widehat{\mathbf{Q}}}_{k^{*}}(i-1)$ and ${\widehat{\mathbf{K}}}_{k^{*}}(i-1)$ represent the $k^{*}$th columns of the matrices $\widehat{\mathbf{Q}}(i-1)$ and $\widehat{\mathbf{K}}(i-1)$, respectively. Therefore, $\widehat{\mathit{\alpha }}\left(i\right)$ in (30) can be calculated as

$$\begin{array}{c}\hfill \widehat{\mathit{\alpha }}\left(i\right)=\widehat{\mathbf{Q}}\left(i\right)\widehat{\mathbf{d}}\left(i\right)=\widehat{\mathit{\alpha }}(i-1)+\frac{(d\left(i\right)-{\widehat{\mathbf{K}}}_{k^{*}}^{\mathrm{T}}(i-1)\widehat{\mathit{\alpha }}(i-1)){\widehat{\mathbf{Q}}}_{k^{*}}(i-1)}{H_i^{-1}\left(i\right)+{\widehat{\mathbf{K}}}_{k^{*}}^{\mathrm{T}}(i-1){\widehat{\mathbf{Q}}}_{k^{*}}(i-1)}.\end{array}$$

(34)

(2) Second Case: dis$\left(\mathbf{u}\right(i),\mathbf{C}(i-1\left)\right)>ϵ$: In this case, we have $\mathbf{C}\left(i\right)=\left\{\mathbf{C}\right(i-1),\mathbf{u}(i\left)\right\}$, $\widehat{\mathsf{\Phi }}\left(i\right)=[\widehat{\mathsf{\Phi }}(i-1),\phi \left(\mathbf{u}\left(i\right)\right)]$, and we have

$$\begin{array}{c}\hfill \widehat{\mathbf{H}}\left(i\right)=\left[\begin{array}{c}\widehat{\mathbf{H}}(i-1)\\ {\mathbf{0}}^{\mathrm{T}}\end{array}\right.\left.\begin{array}{c}\mathbf{0}\\ H_i\left(i\right)\end{array}\right],\widehat{\mathbf{K}}\left(i\right)=\left[\begin{array}{c}\widehat{\mathbf{K}}(i-1)\\ \widehat{\mathbf{h}}{\left(i\right)}^{\mathrm{T}}\end{array}\right.\left.\begin{array}{c}\widehat{\mathbf{h}}\left(i\right)\\ {\kappa }_{ii}\end{array}\right],\end{array}$$

(35)

where $\mathbf{0}$ is the null column vector with a compatible dimension; $\widehat{\mathbf{h}}\left(i\right)=\widehat{\mathsf{\Phi }}{(i-1)}^{\mathrm{T}}\phi \left(\mathbf{u}\left(i\right)\right)$ and ${\kappa }_{ii}={\kappa }_{{\sigma }_1}(\mathbf{u}\left(i\right),\mathbf{u}\left(i\right))$. Combining (31), (35), $\widehat{\mathbf{d}}\left(i\right)={[\widehat{\mathbf{d}}{(i-1)}^{\mathrm{T}},H_i\left(i\right)d\left(i\right)]}^{\mathrm{T}}$, and the block matrix inversion identity [28], we obtain

$$\begin{array}{c}\hfill \widehat{\mathbf{Q}}\left(i\right)=\left[\begin{array}{c}\widehat{\mathbf{Q}}(i-1)+\widehat{r}{\left(i\right)}^{-1}{H}_i\left(i\right){\widehat{\mathbf{z}}}_{\widehat{\mathbf{H}}\left(i\right)}\widehat{\mathbf{z}}{\left(i\right)}^{\mathrm{T}}\\ -\widehat{r}{\left(i\right)}^{-1}{H}_i\left(i\right)\widehat{\mathbf{z}}{\left(i\right)}^{\mathrm{T}}\end{array}\right.\left.\begin{array}{c}-\widehat{r}{\left(i\right)}^{-1}{\widehat{\mathbf{z}}}_{\widehat{\mathbf{H}}\left(i\right)}\\ \widehat{r}{\left(i\right)}^{-1}\end{array}\right],\end{array}$$

(36)

where

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\widehat{\mathbf{z}}}_{\widehat{\mathbf{H}}\left(i\right)}=\widehat{\mathbf{Q}}(i-1)\widehat{\mathbf{H}}(i-1)\widehat{\mathbf{h}}\left(i\right)\hfill \\ \widehat{\mathbf{z}}\left(i\right)=\widehat{\mathbf{Q}}{(i-1)}^{\mathrm{T}}\widehat{\mathbf{h}}\left(i\right)\hfill \\ \widehat{r}\left(i\right)={\kappa }_{{\sigma }_1}(\mathbf{u}\left(i\right),\mathbf{u}\left(i\right))H_i\left(i\right)+{\rho }^i\zeta 2{\sigma }^2/p-H_i\left(i\right)\widehat{\mathbf{h}}{\left(i\right)}^{\mathrm{T}}{\widehat{\mathbf{z}}}_{\widehat{\mathbf{H}}\left(i\right)}\hfill \end{array}\right..\end{array}$$

(37)

Furthermore, due to $\widehat{\mathbf{d}}\left(i\right)={[\widehat{\mathbf{d}}{(i-1)}^{\mathrm{T}},H_i\left(i\right)d\left(i\right)]}^{\mathrm{T}}$, we obtain

$$\begin{array}{c}\hfill \widehat{\mathit{\alpha }}\left(i\right)=\widehat{\mathbf{Q}}\left(i\right)\widehat{\mathbf{d}}\left(i\right)=\left[\begin{array}{c}\widehat{\mathit{\alpha }}(i-1)-\widehat{r}{\left(i\right)}^{-1}{H}_i\left(i\right){\widehat{\mathbf{z}}}_{\widehat{\mathbf{H}}\left(i\right)}(d\left(i\right)-\widehat{\mathbf{h}}{\left(i\right)}^{\mathrm{T}}\widehat{\mathit{\alpha }}(i-1))\\ \widehat{r}{\left(i\right)}^{-1}{H}_i\left(i\right)(d\left(i\right)-\widehat{\mathbf{h}}{\left(i\right)}^{\mathrm{T}}\widehat{\mathit{\alpha }}(i-1))\end{array}\right].\end{array}$$

(38)

The QRMKRP algorithm is summarized in Algorithm 2, where L denotes the dictionary size.

Algorithm 2: The QRMKRP algorithm.

Initialization: $\left\{\mathbf{u}\right(i),d(i\left)\right\},i=1,2,\dots$ $\sigma ,{\sigma }_1,p,\lambda >0,L=1,\mathbf{C}\left(1\right)=\left\{\mathbf{u}\left(1\right)\right\}$, $z\left(1\right)=1-exp\left(-\frac{d^2\left(1\right)}{2{\sigma }^2}\right)$, $H_1\left(1\right)=exp\left(\lambda {\left(z\left(1\right)\right)}^{\frac{p}{2}}\right){\left(z\left(1\right)\right)}^{\frac{p-2}{2}}(1-z\left(1\right)),\widehat{\mathbf{H}}\left(1\right)=\left[H_1\left(1\right)\right]$. $\widehat{\mathbf{Q}}\left(1\right)={\left[\zeta \rho 2{\sigma }^2/p+H_1\left(1\right){\kappa }_{11}\right]}^{-1},\widehat{\mathit{\alpha }}\left(1\right)=\widehat{\mathbf{Q}}\left(1\right)H_1\left(1\right)d\left(1\right)$, $ϵ>0,\rho >0,\zeta \in [0,1]$. Computation: While {$\mathbf{u}\left(i\right),d\left(i\right)\left\}\right(i>1)$ available do   1) Compute the distance between $\mathbf{u}\left(i\right)$ and $\mathbf{C}(i-1)$:    dis$(\mathbf{u}\left(i\right),\mathbf{C}(i-1))=\underset{1\le k\le \left|{\mathbf{C}}_k\left(i-1\right)\right|}{min}{∥\mathbf{u}\left(i\right)-{\mathbf{C}}_k\left(i-1\right)∥}^2$,    where $k^{*}$ = $\underset{1\le k\le \left|{\mathbf{C}}_k\left(i-1\right)\right|}{argmin}{∥\mathbf{u}\left(i\right)-{\mathbf{C}}_k\left(i-1\right)∥}^2$.   2) If dis$\left(\mathbf{u}\right(i),\mathbf{C}(i-1\left)\right)\le ϵ$:    Keep the dictionary unchanged: $\mathbf{C}\left(i\right)=\mathbf{C}(i-1),L\Leftarrow L$,    Update $\widehat{\mathbf{H}}\left(i\right)$ by (32), $\widehat{\mathbf{Q}}\left(i\right)$ by (33), $\widehat{\mathit{\alpha }}\left(i\right)$ by (34).   3) Otherwise:    The dictionary changes: $\mathbf{C}\left(i\right)=\left[\mathbf{C}\right(i-1),\mathbf{u}(i\left)\right],L\Leftarrow L+1$,    Update $\widehat{\mathbf{H}}\left(i\right)$ by (35), $\widehat{\mathbf{Q}}\left(i\right)$ by (36), $\widehat{\mathit{\alpha }}\left(i\right)$ by (38). end while

4. Simulation

In this section, to validate the performance of the proposed RMKRP algorithm and its quantized version, two examples, i.e., Mackey–Glass (MG) chaotic time series prediction and nonlinear system identification, are used to validate the performance superiorities of the proposed two algorithms.

In this example, the noise environment considered is the impulsive noise, which is modeled by the combination of two independent noise processes [17], i.e.,

$$v\left(i\right)=(1-b\left(i\right))v_1\left(i\right)+b\left(i\right)v_2\left(i\right),$$

(39)

where $v_1\left(i\right)$ is an ordinary noise disturbance with small variance and $v_2\left(i\right)$ represents large outliers with large variance; $b\left(i\right)$ is of binary distribution random process over $\{0,1\}$ with $Prob\left\{b\left(i\right)=1\right\}=c$ and $Prob\left\{b\left(i\right)=0\right\}=1-c$ ($0\le c\le 1$ is an occurrence probability). Here, we select $c=0.1$. The distribution of $v_1\left(i\right)$ is considered as a Binary distribution over $\{0.5,-0.5\}$ with probability mass $Prob\left\{x=0.5\right\}=Prob\left\{x=-0.5\right\}=0.5$. In addition, $v_2\left(i\right)$ is modeled by the $\alpha$-stable process, owing to its heavy-tailed probability density function. The $\alpha$-stable process is described by the following characteristic function [29]:

$$f\left(t\right)=exp\left\{j\delta t-\gamma {\left|t\right|}^{\alpha }\left[1+j\beta \mathrm{sgn}\left(t\right)S(t,\alpha )\right]\right\},$$

(40)

where

$$\begin{array}{c}\hfill S(t,\alpha )=\left\{\begin{array}{c}tan\frac{\alpha \pi }{2},\mathrm{if}\alpha \ne 1\\ \frac{2}{\pi }log\left|t\right|,\mathrm{if}\alpha =1\end{array}\right.,\end{array}$$

(41)

with $\alpha \in (0,2]$ being the characteristic factor, $\beta \in [-1,1]$ being the symmetry parameter, $\gamma >0$ being the dispersion parameter, $\mathrm{sgn}(.)$ denotes the sign function, $j=\sqrt{-1}$, and $-\infty <\delta <\infty$ being the location parameter. Generally, a smaller $\alpha$ generates a heavier tail and a smaller $\gamma$ generates fewer large outliers. The characteristic function denoted as $V_{\alpha -\mathrm{stable}}(\alpha ,\beta ,\gamma ,\delta )$ is chosen as $V_{\alpha -\mathrm{stable}}(0.8,0,0.1,0)$ to model the impulse noise in the simulations.

4.1. Chaotic Time Series Prediction

The MG chaotic time series is generated from the following differential equation [9]:

$$\frac{dx\left(t\right)}{dt}=\frac{\beta x(t-\tau )}{1+x{(t-\tau )}^n}-\gamma x\left(t\right),$$

(42)

where $\beta ,\gamma ,n>0$. Here, we set $\beta =0.2$, $\gamma =0.1$, and $\tau =30$. The time series is discretized at a sampling period of six seconds. The training set includes a segment of 2000 samples corrupted by the additive noises which are shown in (39), and another 200 samples without noise are used as the testing set. The kernel size ${\sigma }_1$ in the Gaussian kernel is set to 1. The filter length is set at $L=7$, which means that $[x_t,x_{t-1},\dots ,x_{t-6}]$ is used to predict $x_{t+1}$.

To evaluate the filtering accuracy, the testing mean square error (MSE) is defined as follows:

$$\mathrm{MSE}\left(\mathrm{dB}\right)=\frac{1}{N}10\mathrm{lo}{\mathrm{g}}_{10}({\displaystyle \sum _{i=1}^N}{(d\left(i\right)-\widehat{f}\left(i\right))}^2),$$

(43)

where $\widehat{f}\left(i\right)$ is the estimate of $d\left(i\right)$, and N is the length of testing data.

The KLMS [6], KMCC [22], MKRL [26], KRMC [21], and KRLS [8] algorithms are chosen for performance comparison with RMKRP thanks to their excellent filtering performance. The other sparsification algorithms, i.e., the QKLMS [12], QKMCC [30], QMKRL [26], QKRLS [13], and KRMC-NC [21] algorithms are used for performance comparison with QRMKRP owing to their modest space complexities and excellent performance. All simulation results are averaged over 50 independent Monte Carlo runs.

Since power parameter p, risk-sensitive parameter $\lambda$, and kernel width $\sigma$ are crucial parameters in the proposed RMKRP and QRMKRP algorithms, the influence of these parameters on the performance is first discussed. In the simulations, we take 12 points evenly in the close interval $p\in [1,6]$ and $\sigma \in [0.17,5]$, respectively. The influence of p on the steady-state performance of RMKRP is shown in Figure 2a, where the steady-state MSEs are obtained as averages over the last 100 iterations. The parameters are set as: p is set within $[1,6]$; risk-sensitive parameter $\lambda$ in the KRP is set as 1; $\zeta =0.1$ and $\rho =1$; kernel size $\sigma$ in the KRP is set as 1. As can be seen from Figure 2a, we have that the filtering accuracy of RMKRP is the highest when $p=4$ and decreases gradually when p is either too small or too large. Then, the influence of $\sigma$ on the filtering performance of RMKRP with $p=4$ is shown in Figure 2b, where the steady-state MSEs are obtained as averages over the last 100 iterations. The parameters are set as: risk-sensitive parameter $\lambda$ is fixed at 1; $\sigma$ lies in $[0.17,5]$. From Figure 2b, we see that RMKRP can achieve the highest filtering accuracy when $\sigma$ is about 1. It is reasonable to note that RMKRP are sensitive to outliers when the kernel width is large, and decreases its ability of error-correction when the kernel width is small. Finally, the influence of $\lambda$ on the filtering performance of RMKRP with $\sigma =1$ and $p=4$ is shown in Figure 2c, where the steady-state MSEs are obtained as averages over the last 100 iterations. The parameters are set as: the range of $\lambda$ is selected as $\lambda \in \{0.001,0.01,0.1,0.2,0.3,0.4,0.5,0.6,1,2,3,4\}$. From Figure 2c, we see that $\lambda$ has a slight influence on the filtering accuracy when $\lambda$ is small, and a large $\lambda$ can increase the steady-state MSE obviously. Therefore, from Figure 2, the parameters of RMKRP can be chosen by trials to obtain the best performance in practice. Similarly, the parameters of QRMKRP can be chosen by the same method as that in RMKRP.

Figure 2. Steady-state MSE of RMKRP with different p in MG time series prediction (a); steady-state MSE of RMKRP with different $\sigma$ in MG time series prediction (b); steady-state MSE of RMKRP with different $\lambda$ in MG time series prediction (c).

The performance comparison of QKLMS, QKMCC, QMKRL, KLMS, KMCC, MKRL, KRLS, KRMC, KRMC-NC, and QKRLS is conducted in the same environments as in (39). The parameters of the proposed algorithms are selected by trials to achieve desirable performance, and the parameters of compared algorithms are chosen such that they have almost the same convergence rate. $\lambda =1$, $p=4$, and $\sigma =1$ are set for RMKRP; $\lambda =1$, $p=4$, $\sigma =1$, and $ϵ=0.2$ for QRMKRP; $\eta =0.1$ for KLMS; $\eta =0.09$ and $\sigma =3.5$ for KMCC; $\eta =0.09$, $\sigma =1$, and $\lambda =1$ for MKRL; $\eta =0.1$ and $ϵ=0.2$ for QKLMS; $\eta =0.09$, $ϵ=0.2$, and $\sigma =3.5$ for QKMCC; $\eta =0.09$, $ϵ=0.2$, $\sigma =1$, and $\lambda =1$ for QMKRL; $\zeta =0.1$, $\rho =1$, and $\sigma =3.5$ for KRMC; the novelty criterion thresholds ${\delta }_1=0.15$, ${\delta }_2=0.1$, $\zeta =0.1$, $\rho =1$, and $\sigma =3.5$ for KRMC-NC; $\zeta =0.1$ for KRLS; $\zeta =0.1$ and $ϵ=0.2$ for QKRLS. Figure 3 shows the compared MSEs of RMKRP, QRMKRP, and the compared algorithms. As can be seen from Figure 3, RMKRP achieves a better filtering accuracy than KRLS, KRMC, KLMS, KMCC, and MKRL. QRMKRP achieves a better steady-state testing MSE than the sparsification algorithms including QKRLS, KRMC-NC, QKLMS, QKMCC, and QMKRL. We also see from Figure 3 that the proposed algorithms provide good robustness to impulsive noises. For detailed comparison, the dictionary size, consumed time, and steady-state MSEs in Figure 3 are shown in Table 1. Note that the steady-state MSEs of KLMS, QKLMS, KRLS, and QKRLS are not shown in Table 1 since they cannot converge in such impulsive noise environment. From Table 1, we see that RMKRP has similar consumed time to KRLS and KRMC but provides better filtering accuracy. In addition, QRMKRP provides the highest filtering accuracy in all the compared sparsification algorithms and approaches the filtering accuracy of RMKRP with a significantly lower network size.

Figure 3. Comparison of the MSEs of KLMS, KMCC, MKRL, KRLS, KRMC, and RMKRP in MG time series prediction (a); comparison of the MSEs of QKLMS, QKMCC, QMKRL, QKRLS, KRMC-NC, and QRMKRP in MG time series prediction (b).

Table 1. Simulation results of QKLMS, QKMCC, QMKRL, QKRLS, KRMC-NC, KLMS, KMCC, MKRL, KRLS, KRMC, RMKRP, and QRMKRP in MG time series prediction.

Algorithms	Size	Time (s)	MSE (dB)
KLMS [6]	2000	30.9501 s	N/A
QKLMS [12]	28	2.1011 s	N/A
KRLS [8]	2000	58.5358 s	N/A
QKRLS [13]	28	2.3374 s	N/A
KMCC [22]	2000	30.8285 s	−18.5063
QKMCC [30]	28	2.0995 s	−17.8707
MKRL [26]	2000	30.9117 s	−18.7312
QMKRL [26]	28	2.1063 s	−18.1037
KRMC [21]	2000	58.1229 s	−25.1618
KRMC-NC [21]	462	2.8045 s	−21.5183
QRMKRP	28	2.3443 s	−24.9326
RMKRP	2000	58.2196 s	−28.1802

4.2. Nonlinear System Identification

To further validate the performance superiorities of the proposed RMKRP and QRMKRP algorithms, the nonlinear system identification is considered. Here, the nonlinear system is of the following form [31].

$$\begin{array}{cc}\hfill \mathrm{s}\left(t\right)& =s(t-1)(0.8-0.5exp(-s^2(t-1)))-(0.3+0.9exp(-s^2(t-1)))s(t-2)+0.1sin\left(s(t-1)\pi \right),\hfill \end{array}$$

(44)

where $\mathrm{s}\left(t\right)$ denotes the output at discrete time t with the initial $s(-1)=0.1$ and $s(-2)=0.1$. The two previous outputs $\mathbf{u}\left(k\right)={[s(t-1),s(t-2)]}^{\mathrm{T}}$ are utilized as the input to estimate the current output $\mathrm{s}\left(t\right)$. The training set includes a segment of 2000 samples corrupted by the additive noises shown in (39), and another 200 samples without noise are used as the testing set. The kernel width ${\sigma }_1$ is set to 1 for the Gaussian function. All simulation results are averaged over 50 independent Monte Carlo runs.

Similar to MG chaotic time series prediction, the influence of power parameter p, risk-sensitive parameter $\lambda$, and kernel width $\sigma$ on the performance of RMKRP is also discussed in nonlinear system identification. The influence of p on the steady-state performance of RMKRP is shown in Figure 4a, where the steady-state MSEs are obtained as averages over the last 100 iterations. The parameters are set as: p is set within $[1,6]$; $\lambda$ is set as $0.1$; $\zeta =0.1$ and $\rho =1$; kernel size $\sigma$ in the KRP is set as 1. The influence of $\sigma$ on the filtering performance of RMKRP is shown in Figure 4b, where risk-sensitive parameter $\lambda$ is fixed at $0.1$; $\sigma$ lies in $[0.17,5]$; p is set as 4. The influence of $\lambda$ on the filtering performance of RMKRP is shown in Figure 4c, where the range of $\lambda$ is selected as $\lambda \in \{0.001,0.01,0.1,0.2,0.3,0.4,0.5,0.6,1,2,3,4\}$; $\sigma$ is set as 1; p is set as 4. As can be seen from Figure 4, we can obtain the same conclusions as those in Figure 2.

Figure 4. Steady-state MSE of RMKRP with different p in nonlinear system identification (a); steady-state MSE of RMKRP with different $\sigma$ in nonlinear system identification (b); steady-state MSE of RMKRP with different $\lambda$ in nonlinear system identification (c).

We compare the filtering performance of QKLMS, QKMCC, QMKRL, KLMS, KMCC, MKRL, KRLS, KRMC, KRMC-NC, and QKRLS in the same environments as in (39). The parameters of the proposed algorithms are selected by trials to achieve desirable performance, and the parameters of compared algorithms are chosen such that they have almost the same convergence rate. $\lambda =0.1$, $p=4$, and $\sigma =1$ are set for RMKRP; $\lambda =0.1$, $p=4$, $\sigma =1$, and $ϵ=0.2$ for QRMKRP; $\eta =0.1$ for KLMS; $\eta =0.09$ and $\sigma =3.5$ for KMCC; $\eta =0.09$, $\sigma =1$, and $\lambda =2$ for MKRL; $\eta =0.1$ and $ϵ=0.2$ for QKLMS; $\eta =0.09$, $ϵ=0.2$, and $\sigma =3.5$ for QKMCC; $\eta =0.09$, $ϵ=0.2$, $\sigma =1$, and $\lambda =2$ for QMKRL; $\zeta =0.1$, $\rho =1$, and $\sigma =3.5$ for KRMC; the novelty criterion thresholds ${\delta }_1=0.01$, ${\delta }_2=0.1$, $\zeta =0.1$, $\rho =1$, and $\sigma =3.5$ for KRMC-NC; $\zeta =0.1$ for KRLS; $\zeta =0.1$ and $ϵ=0.2$ for QKRLS. Figure 5 shows the compared MSEs of RMKRP, QRMKRP, and the compared algorithms. For detailed comparison, the dictionary size, consumed time, and steady-state MSEs in Figure 5 are also shown in Table 2, where the steady-state MSEs of KLMS, QKLMS, KRLS, and QKRLS are not shown since they cannot converge in such impulsive noise environments. From Figure 5 and Table 2, we can obtain the same conclusions as those in Figure 3 and Table 1.

Figure 5. Comparison of the MSEs of KLMS, KMCC, MKRL, KRLS, KRMC, and RMKRP in nonlinear system identification (a); comparison of the MSEs of QKLMS, QKMCC, QMKRL, QKRLS, KRMC-NC, and QRMKRP nonlinear system identification (b).

Table 2. Simulation results of QKLMS, QKMCC, QMKRL, QKRLS, KRMC-NC, KLMS, KMCC, MKRL, KRLS, KRMC, RMKRP, and QRMKRP in nonlinear system identification.

Algorithms	Size	Time (s)	MSE (dB)
KLMS [6]	2000	21.2447 s	N/A
QKLMS [12]	14	1.7284 s	N/A
KRLS [8]	2000	48.6055 s	N/A
QKRLS [13]	14	1.9643 s	N/A
KMCC [22]	2000	21.1328 s	−19.233
QKMCC [30]	14	1.763 s	−17.9723
MKRL [26]	2000	21.0313 s	−19.5390
QMKRL [26]	14	1.7243 s	−18.5748
KRMC [21]	2000	48.7601 s	−28.7583
KRMC-NC [21]	496	2.6874 s	−23.671
QRMKRP	14	1.9681 s	−27.3128
RMKRP	2000	48.6101 s	−34.0790

5. Conclusions

In this paper, the kernel risk-sensitive mean p-power error (KRP) criterion is proposed by constructing mean p-power error (MPE) into kernel risk-sensitive loss (KRL) in RKHS, and some basic properties are presented. The KRP criterion with power parameter p is more flexible than KRL to handle the signal corrupted by impulsive noises. Two kernel recursive adaptive algorithms are derived to obtain desirable filtering accuracy under the minimum KRP (MKRP) criterion, i.e., the recursive minimum KRP (RMKRP) and quantized RMKRP (QRMKRP) algorithms. The RMKRP can achieve higher accuracy but with almost identical computational complexity as that of the KRLS and KRMC. The vector quantization method is introduced into RMKRP, thus generating QRMKRP, and QRMKRP can effectively reduce network size while maintaining the filtering accuracy. Simulations conducted in Mackey–Glass (MG) chaotic time series prediction and nonlinear system identification under impulsive noises illustrate the superiorities of RMKRP and QRMKRP from the aspects of robustness and filtering accuracy.