On Optimality and Robustness in Linear Dynamic System Identification

Abstract

Strong consistency and asymptotic error distribution for a new class of nonlinear recursive parameter estimation algorithms of an approximate Newton–Raphson type are established. The system model is given in the discrete-time domain by a linear difference equation with constant parameters. The parameter estimator design is based on martingale theory and the Cramér–Rao (CR) theorem, providing a maximum likelihood (ML)-type optimal recursive parameter identification algorithm, whose minimum asymptotic estimation error covariance matrix achieves the CR bound under the worst-case pdf within a specified class; this worst-case pdf simultaneously yields the maximum asymptotic error covariance matrix within the class. However, the worst-case pdf does not generally exist, making the min–max optimal design indeterminable. Therefore, an approximate ML (AML)-type optimal on a class design, based on a suboptimal worst-case pdf, minimizing the scalar Fisher information within the specified class, has also been developed. The proposed design minimizes the conditional estimation error covariance under the specified suboptimal worst-case pdf. Such an approach results in Huber’s M-robustified version of the AML-based optimal design on the class of contaminated Gaussian pdfs. The practical performance of the proposed approach is analyzed through statistical validation, based on the relative asymptotic estimation efficiency measure and Monte Carlo simulations.

1. Introduction

Modeling is an important activity in various engineering and scientific fields, involving different types of applications, some of which are technical (mechanical, chemical, electrical and hydraulic plants), while others involve biological, medical, ecological, economic and psychological processes, etc. [1,2,3,4,5,6,7,8]. In general, there exist two types of modeling approaches. The first one assumes enough knowledge about the physical phenomenology of the process to incorporate a few idealized physical laws to which the process dynamics is confined, combined with an adequate representation of the noisy measurements. Since a such model is based on a lumped physical phenomenology characterized by the state-space structure, it is called a white-box or state-space model [6,7,8,9,10]. The second one assumes that adequate information about the process to be handled is not available, leading to the need to derive its model solely from input–output data. Since this model is based on a parametric, or polynomial, input–output structure and reflects the absence of process knowledge, it is called a black-box or parametric polynomial input–output model [11,12,13,14,15,16,17]. In this sense, one has to define an experiment to obtain information about the process in question. Furthermore, the design of the experiment must be followed by a step for parameter identification. In general, input–output representations can be used easily for process analysis, simulations and control purposes, while state-space representations may be generalized easily.

Parametric estimation methods have become a cornerstone of modern signal processing [6], with applications ranging from speech processing [18,19,20] and adaptive filtering [21,22,23] to time-series analysis, parameter and state estimation, pattern classification and control [6,7,8,9,10,11,12,13,14,15,16,17]. The works of Ljung, Goodwin, Tsypkin and others provide both theoretical and practical frameworks for the design and performance evaluation of such methods [11,12,13,14,15,24,25]. Parametric methods may be viewed as model-based procedures in which, once the model set is selected, the unknown parameters are estimated from measurement data according to an adopted optimization criterion. When the signal statistics are not exactly known, adaptive techniques based on numerical optimization theory [26,27,28,29,30,31] offer a solution: for stationary signals, the adaptive estimates converge to the optimal ones [6,7,8,11,12,13,14,15,16,17]. The main problem is then the choice of an adaptation algorithm—gradient, Newton–Raphson, etc.—that iteratively adjusts the required parameters (statistics, gains, coefficients) under incomplete prior knowledge. In this context, the quadratic score is optimal in the MMSE sense under Gaussian disturbances, leading to the standard least squares (LS) identification algorithms [2,3,4,5,6,7,8,11,12,13,14,15,16,17]. However, the ML optimal estimation algorithms, as well as adaptive techniques, which rely on the assumption of Gaussian disturbances, are particularly inefficient when dealing with heavy-tailed, approximately Gaussian-distributed observations, which can lead to rare but huge outliers. In this sense, owing to incomplete measurements and communication errors, most industrial and scientific data are contaminated Gaussian-distributed, producing observation and innovation outliers. In addition, structural outliers may occur due to inadequate mathematical models or computational errors [32,33,34,35]. Consequently, investigations have focused on developing robust parameter estimators that effectively suppress the influence of multiple outliers in practice. In robust statistics, such a feature of the estimator is known as resistance robustness. Moreover, owing to the central limit theorem of statistics, it is also desirable that the robust estimator exhibit good performance under both purely Gaussian and heavy-tailed Gaussian measurements, resulting in the efficiency robustness feature. Practical robustness involves both the efficiency and resistance features, making it appealing to engineers and practitioners [32,33,34,35]. Huber’s approximate ML (AML), or M-robust, concept is frequently used to design practically robust parameter estimates in various applications [32,33,34,35,36,37,38,39].

The other two robustness concepts, namely min–max and qualitative robustness, are based on rigorous mathematical approaches and are thus more appealing to scientists and theoretical workers. In this sense, qualitative robustness involves the definition of Hampel’s influence function as an efficient tool that enables a robust estimator to effectively eliminate the influence of multiple outliers [39]. In addition, Huber’s min–max robust statistical approach minimizes the maximum anticipated asymptotic estimation error attained at the least favorable, or worst-case, distribution within the specified class [35]. This, in turn, provides the maximum asymptotic rate of the estimates’ convergence under the worst-case observation noise distribution. However, the matrix optimization problem is a nonclassical variational task, so its solution does not generally exist [26,27,28]. In particular, for static regression plants or linear finite impulse response (FIR) dynamic systems, the worst-case pdf is derived from the minimization of the scalar Fisher information within the specified class [15,35,38]. In addition, most robust estimation schemes are minimization procedures, which leads to highly nonlinear parameter estimation algorithms [32,33,34,35,36,37,38,39]. Furthermore, based on the computational requirements of online or real-time applications, one can choose to calculate the estimate recursively rather than nonrecursively in a batch processing mode [1,2,3,4,5,6,7,8,11,12,13,14,15,16,17].

In relation to the existing literature, the proposed approach differs from closely related methods in the following aspects. Classical recursive ML and prediction-error identification schemes [11,12,13,14,15,16,17] rely on the quadratic score, optimal under Gaussian disturbances, and their robust modifications are typically introduced heuristically, without an explicit optimality framework. Huber’s M-robust estimation theory [35,39], as well as the min–max optimal identification of Poljak and Tsypkin [15,24,25], provides such a framework, but the exact worst-case solution is available only for static regression and FIR structures. Robust recursive Newton-type algorithms, e.g., [21,22,23,24,25,38], address the ARX/IIR case, but complete asymptotic characterization is lacking. The contributions of this paper are therefore threefold: (i) the strong consistency and asymptotic normality of a class of approximate Newton–Raphson recursive estimators with a general admissible influence function are rigorously established by martingale theory (Theorems 1 and 2); (ii) it is shown that the exact min–max optimal design is generally indeterminable for IIR dynamic systems, which motivates the proposed AML-based optimal design on a class, minimizing at each stage the conditional estimation error covariance under a suboptimal worst-case pdf that minimizes the scalar Fisher information; (iii) the link between the optimal nonlinearity and the gain matrix recursion is established through the statistical linearization coefficient, for which two realizable approximations (fixed and variable factors) are proposed and comparatively analyzed.

In this paper, a new class of nonlinear recursive estimation algorithms of an approximate Newton–Raphson type for the parameter identification of linear discrete time-invariant dynamic stochastic systems is considered. The manuscript is organized in the following manner. The introductory Section 1 presents a brief discussion of frequently used approaches to dynamic system modeling, involving system parameter identification and robust estimation concepts. Section 2 is devoted to designing a class of approximate Newton–Raphson-type recursive parameter estimation algorithms under the exactly known observation noise statistics. Conditions under which the derived identification algorithm converges with probability one (w.p.1), or almost surely (a.s), are discussed in Section 3. In addition, the asymptotic estimation error analysis, involving the convergence of the asymptotic estimation error to a zero-mean Gaussian pdf, is presented in Section 4. The problem of designing a feasible recursive nonlinear M-robustified version of the AML-based optimal parameter estimation algorithm on a class of pdfs is considered in Section 5. Section 6 contains the implementation considerations and the statistical linearization analysis, based on the relative estimation efficiency quantitative measure. The performance analysis is carried out for the practically important class of Gaussian-distributed observations corrupted by impulsive noise or outliers. Section 7 presents Monte Carlo simulation results that illustrate the theoretical analysis from Section 6, as well as investigating the ability of the proposed parameter identification procedure to achieve practical robustness, involving both the resistance and efficiency robustness characteristics. Concluding remarks are presented in Section 8. The detailed derivation of the asymptotic estimation error zero-mean normal distribution, using the martingale theory results, is presented in Appendix A.

2. Problem Formulation

An abstract single-input single-output (SISO) linear discrete time-invariant dynamic stochastic system is described by a linear difference equation with constant parameters:

$$y\left(i\right)=-{\displaystyle \sum _{k=1}^n{a}_k^{*}y\left(i-k\right)+{\displaystyle \sum _{k=1}^m{b}_k^{*}u\left(i-k\right)+e\left(i\right)}}$$

(1)

Here, $y\left(i\right)$ is the output signal sample, $u\left(i\right)$ is the input signal sample, and $e\left(i\right)$ is the random disturbance, or observation noise, sample at the discrete time indexed by $i$. In normal operation, the observed input–output sequences $\left\{u\left(i\right)\right\}$ and $\left\{y\left(i\right)\right\}$ are wide-sense stationary random discrete-time processes [9,10,11,12,13,14,15,16,17]. The random noise samples $e\left(i\right)$ are zero-mean, independent and identically distributed (i.i.d.) scalar real random variables. Introducing the shift operator $q^{-k}x\left(i\right)=x\left(i-k\right)$, one can rewrite (1) in the form

$$A\left(q^{-1}\right)y\left(i\right)=B\left(q^{-1}\right)u\left(i\right)+e\left(i\right).$$

(2)

where the characteristic polynomial $A\left(\cdot \right)$ and control polynomial $B\left(\cdot \right)$ are given by

$$A\left(q^{-1}\right)=1+{\displaystyle \sum _{k=1}^n{a}_k{q}^{-k}};\hspace{1em}B\left(q^{-1}\right)={\displaystyle \sum _{k=1}^m{b}_k{q}^{-k}}$$

(3)

assuming that the polynomial orders $m$ and $n$ are given in advance. In addition, relation (1) can also be rewritten in the linear regression form

$$y\left(i\right)={{\theta }_{*}}^{T}Z\left(i\right)+e\left(i\right);\hspace{1em}{{\theta }_{*}}^T=\left\{a_1^{*},\dots ,a_n^{*},b_1^{*},\dots ,b_m^{*}\right\}$$

(4)

where ${\theta }_{*}$ is the true $\left(n+m\right)\times 1$ column vector of unknown system parameters to be estimated, while $Z\left(i\right)$ is the information, or regression, $\left(n+m\right)\times 1$ column vector

$$Z^T\left(i\right)=\left\{-y\left(i-1\right),\dots ,-y\left(i-n\right),u\left(i-1\right),\dots ,u\left(i-m\right)\right\}$$

(5)

Since the disturbance or noise sequence $\left\{e\left(i\right)\right\}$ directly enters relation (2), this representation belongs to the family of equation errors (EEs), in which the EE is represented by a white real scalar discrete stochastic process [14]. Representation (2) is also called an autoregressive with exogenous input (ARX) model [5,6,7,8,9,11,12,13,14,15,16,17]. In addition, model (2) is characterized by an infinite impulse response (IIR) under the unit $\delta$-impulse excitation. In particular, if the polynomial $A\left(\cdot \right)=1$ in (3), then representation (2) is known as a finite impulse response (FIR) model [2,3,4,5,6,7,8,11,12,13,14,15,16,17].

The problem of the recursive identification of the system described by (4) is considered as the task of estimating in real time the unknown parameter vector ${\theta }_{*}$ by utilizing the current input–output observations $\left\{u\left(i\right),y\left(i\right)\right\}$. The formulation of the identification problem reduces to the choice of prediction model $\widehat{y}\left(i|\theta \right)$, where ${\theta }^T=\left\{a_1,\dots ,a_n,b_1,\dots ,b_m\right\}$ is the given vector of model parameters, as well the choice of an identification performance criterion, or quality measure, to be minimized,

$$J\left(\theta \right)=E\left\{F\left[\epsilon \left(i,\theta \right)\right]\right\}; \epsilon \left(i,\theta \right)=y\left(i\right)-\widehat{y}\left(i|\theta \right)$$

(6)

where $E\left\{\cdot \right\}$ is the mathematical expectation. Here, $\epsilon \left(\cdot \right)$ is the prediction error, also known as the measurement residual or innovation. Moreover, $F:\mathbb{R}\to \mathbb{R}$ is assumed to belong to the class $\mathrm{F}$ of admissible score functions, where $F\in \mathrm{F}$ if $F$ is real-valued, symmetric, $F\left(z\right)=F\left(-z\right)$, convex, $F\left(0\right)=0$, and absolutely continuous with derivative $\psi \left(z\right)=F^{\prime }\left(z\right)$ existing for almost all $z\in \mathbb{R}$ [5,6,7,8,11,12,13,14,15,16,17,21,22,23,24,25]. The solution of the posed optimization problem in (6) reduces to choosing an online parameter identification algorithm

$$\widehat{\theta }\left(i\right)=f\left[\widehat{\theta }\left(i-1\right),u\left(i\right),y\left(i\right)\right]$$

(7)

which recursively calculates the current vector estimate $\widehat{\theta }\left(i\right)$ of the unknown parameter vector ${\theta }_{*}$ in (4) from the preceding parameter vector estimate $\widehat{\theta }\left(i-1\right)$ and the current input–output observations $\left\{u\left(i\right),y\left(i\right)\right\}.$ Therefore, the aim is to determine optimally, in some sense, the prediction model $\widehat{y}\left(\cdot |\cdot \right)$ and the score function $F\left(\cdot \right)$ in (6), as well as the estimation procedure $f\left(\cdot \right)$ in (7). Here, prior information about the system defines an optimal prediction model in the MMSE sense. Thus, taking into account (2)–(5), and utilizing MMSE criterion $E\left\{{\epsilon }^2\left(i,\theta \right)\right\}$, one gets the prediction model [5,6,7,8,11,12,13,14,15,16,17]

$$\widehat{y}\left(i|\theta \right)=\left[1-A\left(q^{-1}\right)\right]y\left(i\right)+B\left(q^{-1}\right)u\left(i\right)={\theta }^{T}Z\left(i\right)$$

(8)

Let the i.i.d. disturbances $\left\{e\left(i\right)\right\}$ in (2) admit a probability density function $p:\mathbb{R}\to {\mathbb{R}}_{+}$, ${\displaystyle \int \mathbb{R} p\left(z\right)dz=1}$, which is assumed to be symmetric, i.e., $p\left(z\right)=p\left(-z\right)$, strictly positive and absolutely continuous on $\mathbb{R}$, with finite Fisher information $I\left(p\right)={{\displaystyle \int \left(p^{\prime }/p\right)}}^{2}pdz<\infty$ [9,10]. Under these conditions, the ML-based optimal score function and the associated influence function are well defined and given by [5,6,7,8,9,10,11,12,13,14,15,16,17,18]

$$F\left(z\right)=F_0\left(z\right)=-\log p\left(z\right); \psi \left(z\right)={\psi }_0\left(z\right)={\displaystyle \frac{d{F}_0\left(z\right)}{dz}}={F^{\prime }}_0\left(z\right)$$

(9)

To derive the optimal recursive identification algorithm $f\left(\cdot \right)$ in (7), let us also consider the empirical average loss

$$J_i\left(\theta \right)=i^{-1}{\displaystyle \sum _{k=1}^{i}F\left[\epsilon \left(k,\theta \right)\right]}$$

(10)

Under certain, mild conditions, the empirical criterion $J_i\left(\theta \right)$ in (10) asymptotically converges to the optimal criterion $J\left(\theta \right)$ in (6) [15,24,25]. The empirical criterion in (10) can be minimized recursively using an approximate Newton–Raphson-based iterative numerical algorithm:

$$\widehat{\theta }\left(i\right)=\widehat{\theta }\left(i-1\right)-{\left[i{\nabla }_{\theta }^2{J}_i\left(\widehat{\theta }\left(i-1\right)\right)\right]}^{-1}\left[i{\nabla }_{\theta }{J}_i\left(\widehat{\theta }\left(i-1\right)\right)\right]; \widehat{\theta }\left(0\right)={\theta }_0$$

(11)

where ${\nabla }_{\theta }\left(\cdot \right)=\partial \left(\cdot \right)/\partial \theta ={\left\{\partial \left(\cdot \right)/\partial a_1,\dots ,\partial \left(\cdot \right)/\partial b_m\right\}}^T$ is the gradient vector, or the partial derivative operator [15,24,25,31]. Moreover, for a large enough time index $i$, the optimality condition implies that the gradient vector is close to zero, i.e., ${\nabla }_{\theta }{J}_{i-1}\left(\theta \right)\approx 0$, and one obtains

$$\begin{array}{l}i{\nabla }_{\theta }{J}_i\left(\widehat{\theta }\left(i-1\right)\right)=-Z\left(i\right)\psi \left[\epsilon \left(i,\widehat{\theta }\left(i-1\right)\right)\right]; \psi \left(\cdot \right)=F^{\prime }\left(\cdot \right)\\ i{\nabla }_{\theta }^2{J}_i\left(\widehat{\theta }\left(i-1\right)\right)=\alpha {\displaystyle \sum _{k=1}^{i}Z\left(k\right)Z^T\left(k\right)}; \alpha =E_p\left\{{\psi }^{\prime }\left[e\left(\cdot \right)\right]\right\}\end{array}$$

(12)

with $E_p\left\{\cdot \right\}$ being the expectation with respect to the pdf $p$ of the i.i.d. random variable $e\left(i\right)$ in (2). In addition, let us introduce the current gain matrix in the $i$-th discrete-time stage:

$$\begin{array}{ll}\Gamma \left(i\right)& ={\left[i{\nabla }_{\theta }^2{J}_i\left(\widehat{\theta }\left(i-1\right)\right)\right]}^{-1}\\ & ={\left[\alpha {\displaystyle \sum _{k=1}^{i-1}Z\left(k\right)Z^T\left(k\right)}+\alpha Z\left(i\right)Z^T\left(i\right)\right]}^{-1}={\left[\Gamma \left(i-1\right)+\alpha Z\left(i\right)Z^T\left(i\right)\right]}^{-1}\end{array}$$

(13)

Taking into account the matrix inversion lemma, one gets, from (11) and (12), the parameter identification algorithm

$$\widehat{\theta }\left(i\right)=\widehat{\theta }\left(i-1\right)+\Gamma \left(i\right)Z\left(i\right)\psi \left[\epsilon \left(i,\widehat{\theta }\left(i-1\right)\right)\right]; \widehat{\theta }\left(0\right)=0$$

(14)

$$\epsilon \left(i,\widehat{\theta }\left(i-1\right)\right)=y\left(i\right)-Z^T\left(i\right)\widehat{\theta }\left(i-1\right); \alpha =E_p\left\{{\psi }^{\prime }\left[e\left(\cdot \right)\right]\right\}$$

(15)

$$\Gamma \left(i\right)=\Gamma \left(i-1\right)-{\displaystyle \frac{\Gamma \left(i-1\right)Z\left(i\right)Z^T\left(i\right)\Gamma \left(i-1\right)}{{\alpha }^{-1}+Z^T\left(i\right)\Gamma \left(i-1\right)Z\left(i\right)}}; \Gamma \left(0\right)=\gamma I$$

(16)

where $I$ is an identity matrix and $\gamma$ is a finite positive constant, while $\alpha$ is the statistical linearization coefficient in (12). In general, prior information about the true parameter vector ${\theta }_{*}$ in (4) defines the initial values of the parameter estimates $\widehat{\theta }\left(0\right)$ and the gain matrix $\Gamma \left(0\right)$ [11,12,13,14,15,16,17,23,24,25]. If such reliable prior information does not exist, the most frequently used initial guesses are those given in relations (14) and (16), namely $\widehat{\theta }\left(0\right)=0$ and $\Gamma \left(0\right)=\gamma I$, respectively.

A special case of the recursive nonlinear algorithm (14)–(16) is the recursive linear least squares (RLS) method, which is optimal, in the MMSE sense, for the Gaussian disturbance pdf $p$. It is widely recognized that, under specific conditions regarding the random disturbances $e\left(i\right)$ in (4), the RLS parameter estimates are consistent [5,6,7,8,11,12,13,14,15,16,17,40,41]. In this context, the asymptotic performance of the nonlinear parameter estimation algorithm (14)–(16) involving a.s or w.p.1 convergence, as well as the convergence of the asymptotic estimation error to a zero-mean normal pdf, is analyzed in the following.

3. Brief Review of Parameter Estimate Consistency

In general, the convergence of recursive stochastic estimation procedures may be analyzed using at least two approaches. The first one is the ordinary differential equation (ODE) approach, based on defining the deterministic ODE associated with the parameter estimation algorithm under consideration, together with the application of the Lyapunov direct method to analyze the ODE stability [11,12,23,38,42]. The second one is based on standard martingale theory [43,44,45]. In this sense, let $\left(\Omega ,\mathrm{F},P\right)$ be a probability space, with ${\mathrm{F}}_1\subset {\mathrm{F}}_2\subset \dots$ being an increasing sequence of sub-$\sigma$-fields $\mathrm{F}$, and let $\left\{x_t\right\}$ be a sequence of real scalar random variables attached to $\left\{{\mathrm{F}}_t\right\}$ [9,10]. Then, the pair $\left\{x_t,{\mathrm{F}}_t\right\}$ represents a martingale if the expectation $E\left\{\left|x_t\right|\right\}<\infty$ and the conditional expectation $E\left\{x_t|{\mathrm{F}}_{t-1}\right\}=x_{t-1}$ w.p.1. Furthermore, provided that $E\left\{x_t|{\mathrm{F}}_{t-1}\right\}\le x_{t-1}$ w.p.1, the pair $\left\{x_t,{\mathrm{F}}_t\right\}$ represents a supermartingale [43,44].

In general, the convergence analysis of the recursive stochastic estimation procedure (14)–(16) can be based on the assumptions concerning the disturbance or noise sequence $\left\{e\left(i\right)\right\}$ in (4); the features of the nonlinear transformation or influence function $\psi \left(\cdot \right)$ in (14) applied to the prediction error, innovation or measurement residual $\epsilon \left(\cdot \right)$ in (15); and the properties of the gain matrix sequence $\left\{\Gamma \left(i\right)\right\}$ in (16). A possible set of assumptions that ensures the strong consistency of the parameter estimate sequence $\left\{\widehat{\theta }\left(i\right)\right\}$ in (14) is summarized in Theorem 1 below.

Theorem 1. 

Let model (4) and the algorithm in (14)–(16) satisfy the following conditions:

C1. 

The stochastic sequence  $\left\{e\left(i\right)\right\}$ consists of i.i.d. real scalar random variables, having a symmetric pdf $p$ with zero mean $E_p\left\{e\left(i\right)\right\}=0$ and finite variance $E_p\left\{e^2\left(i\right)\right\}={\sigma }_e^2<\infty$.

C2. 

The odd real scalar influence function $\psi \left(\cdot \right)$ is continuous almost everywhere.

C3. 

The function $\psi \left(\cdot \right)$ is bounded, satisfying $\left|\psi \left(z\right)\right|\le k_1\left(1+k_2\left|z\right|\right)$ , $0<k_1<\infty$ , $0\le k_2<\infty$.

C4. 

The linearization coefficient $\alpha$ in (12) is a positive and finite constant; $0<\alpha \le k_3<\infty$.

C5. 

There exists a real positive constant ${\delta }_1>0$ such that

$$z\varphi \left(z\right)\ge {\displaystyle \frac{1}{2}}{\delta }_1\alpha z^2;\mathrm{ }\varphi \left(z\right)=E\left\{\psi \left[z-e\left(i\right)\right]|{\mathrm{F}}_{i-1}\right\}$$

with $\left\{{\mathrm{F}}_i\right\}$ being the sequence of increasing sub-$\sigma$-fields, generated by the observations up to present discrete-time $i$ , while the coefficient $\alpha$ is defined by condition C4.

C6. 

If the scalar term $r\left(i\right)=Tr\left\{{\Gamma }^{-1}\left(i\right)\right\}$ , where $Tr\left\{\cdot \right\}$ denotes the matrix trace, then the observation vector $Z\left(i\right)$ in (5) satisfies

$${‖Z\left(i\right)‖}^2\le M{\log }^{{\delta }_2}r\left(i\right); {\delta }_2>0, M\ge 0$$

with $‖\cdot ‖$ being the Euclidean norm.

C7. 

There exists a real constant $c>1$ , such that

$$\underset{i\to \infty }{\lim }{\log }^{c}r\left(i\right)/{\lambda }_{\min }\left\{{\Gamma }^{-1}\left(i\right)\right\}=0 w.p.1\mathit{and} \underset{i\to \infty }{\lim }{\lambda }_{\min }\left\{{\Gamma }^{-1}\left(i\right)\right\}=\infty w.p.1$$

where ${\lambda }_{\min }\left\{\cdot \right\}$ is the minimum eigenvalue of a matrix.

Then, the parameter estimate sequence $\left\{\widehat{\theta }\left(i\right)\right\}$ exhibits strong consistency

$$\underset{i\to \infty }{\lim }\widehat{\theta }\left(i\right)={\theta }_{*} w.p.1,or \mathrm{P}\left\{\underset{i\to \infty }{\lim }\widehat{\theta }\left(i\right)={\theta }_{*}\right\}=1$$

where $P\left\{\cdot \right\}$ denotes the probability of a random event.

The proof of Theorem 1, based on the standard martingale theory results, is presented in [41]. In this context, Neveu’s lemma is the basic martingale convergence result [43].

It is worth noting that the polynomial $A\left(\cdot \right)$ in (2) can be arbitrary, so that the algorithm in (14)–(16) can also be used for the parameter estimation of unstable systems. However, since bounded input–bounded output (BIBO) stability is one of the most desirable properties of a linear discrete time-invariant dynamic system, it is supposed here that the polynomial $A\left(\cdot \right)$ in (2) is a stable one [6,7,8,9,10,11,12,13,14,15,16,17]. Moreover, hypothesis C1 represents a standard whiteness condition on random disturbances or observation noise [9,10]. In other words, assumption C1 results in a zero-mean white observation noise sequence $\left\{e\left(i\right)\right\}$ in (4) with finite variance ${\sigma }^2$. Conditions C2–C5 define a class of admissible nonlinear residual transformations $\psi \left(\cdot \right)$ in (14) of the prediction residuals $\epsilon \left(\cdot \right)$ in (15). In particular, for the linear residual transformation $\psi$, condition C5 reduces to ${\delta }_2<2$. Furthermore, it is commonly adopted that $k_2=0$ in C3, yielding a $\psi$-function bounded by a finite constant. Moreover, under such a bounded $\psi$-function, the noise variance ${\sigma }^2$ in C1 need not be finite. In addition, condition C5 is fulfilled if the function

$$\varphi \left(a\right)={\displaystyle {\displaystyle \underset{0}{\overset{\infty }{\int }}}\left[\psi \left(z+a\right)-\psi \left(z-a\right)\right]dP\left(z\right)}$$

is monotonically nondecreasing [15,23,24,25,38]. Taking into account hypotheses C1 and C2, the last requirement is satisfied if the nonlinear residual transformation $\psi$ and the observation noise probability distribution function $P\left(\cdot \right)$ have a common point of increase, meaning that

$$\psi \left(z+\epsilon \right)>\psi \left(z-\epsilon \right), P\left(z+\epsilon \right)>\psi \left(z-\epsilon \right)$$

for some real scalar $z$-value and every real scalar value $\epsilon >0$. This, in turn, results in $a\varphi \left(a\right)>0$ for $a\ne 0$ and $\varphi \left(0\right)=0$. Therefore, hypothesis C5 is fulfilled if the odd and almost everywhere continuous $\psi$-function is monotonically nondecreasing and piecewise continuously differentiable. As mentioned above, a desirable requirement is the boundedness of the $\psi$-function, with $k_2=0$ in C3. Additionally, even though the polynomial $A\left(\cdot \right)$ in (2) is stable, the underlying stochastic system may have signals that are unbounded [6,7,8,9,10,11,12,13,14,15,16,17]. Hence, hypothesis C6 defines the growth of the observation vector $Z\left(i\right)$ in (5), with $M$ being an arbitrary positive real constant. In particular, the case ${\delta }_2\to 0$ (i.e., ${‖Z\left(i\right)‖}^2\le M$) corresponds to bounded measurement data. Finally, some assumptions on the input data must be imposed to obtain fairly good identification results. Such an input requirement is called the persistent excitation condition [5,6,7,8,11,12,13,14,15,16,17]. In this sense, C7 is a rather weak hypothesis for consistent parameter estimates [40,41]. Taken together, both conditions C6 and C7 ensure that the gain matrix $\Gamma \left(i\right)$ in (16) approaches zero at the proper rate when the time index $i$ increases, i.e., neither too fast nor too slow. In other words, the aim of such an assumption is to ensure that the observations contain sufficient information about the unknown parameter vector ${\theta }_{*}$ in (4). This requirement is physically similar to the concept of stochastic observability, describing the information content of measurement data [9,11,12,13,14,15,16,17].

It is worth emphasizing that not all of conditions C1–C7 carry the same practical weight. Conditions C1 and C3 are mild and easily arranged by design, since they concern the whiteness of the disturbance sequence and the boundedness of the influence function, respectively, both of which are properties of the estimator rather than of the observed data. Conditions C6 and C7, by contrast, concern the growth rate of the observation vector and the persistent excitation of the input and are therefore more likely to be only approximately satisfied over a finite data record in a practical application. However, since C6 and C7 are asymptotic requirements, a transient violation over a limited number of samples does not by itself invalidate the use of the algorithm; it may, at most, slow down the empirical rate of convergence until the observations again contain sufficient information about the parameter vector to be estimated.

In general, in addition to the asymptotic convergence analysis of the recursive nonlinear algorithm (14)–(16), an estimation algorithm must provide a suitable measure of the estimation quality. In this sense, the asymptotic estimation error distribution is analyzed in the following.

4. Asymptotic Estimation Error Distribution

Once the convergence properties of the parameter estimates, (14)–(16), are established, the next natural question is to determine how rapidly the estimates converge to the limit. In particular, if the asymptotic pdf of the estimation error is a zero-mean normal, the mean value and covariance matrix represent the sufficient statistics [9,10]. Thus, one should analyze the convergence rate, as well as the estimator optimality, by utilizing the asymptotic covariance matrix of the estimation error. In this context, the asymptotic estimation error analysis results are presented in the following Theorem 2. Therefore, let us first state a few auxiliary lemmas, which are used in the asymptotic estimation error analysis, together with the standard martingale convergence results [43,44].

Lemma 1 

(Ljung and Söderström) [11]. Let $\left\{b_n\right\}$ be a sequence of scalar real quantities such that $b_n>0$, $\underset{n\to \infty }{\lim }{b}_n=0$ and $n{b}_n<c\left({\displaystyle \sum _{k=1}^{n-1}{b}_k^2}+n^{{\alpha }_1}\right)$ for some real constant  $c>0$ and ${\alpha }_1\in \left(0,1\right)$ . Then,

$${\displaystyle \sum _{k=1}^{n-1}{b}_k^2}<c{n}^{\beta }; \beta =\left\{\begin{array}{ll}\max \left(0,2{\alpha }_1-1\right)\hfill & \mathrm{for} {\alpha }_1\ne 1/2\hfill \\ \epsilon , \left(\epsilon >0\right)\hfill & \mathrm{for} {\alpha }_1=1/2\hfill \end{array}\right.$$

Lemma 2 

(Hall and Heyde) [44]. Let $\left\{x\left(t\right)\right\}$ be a scalar real discrete stochastic process, and let the sequence $L_n={\displaystyle \sum _{t=1}^{n}x\left(t\right)}$ converge w.p.1 to $L$, with the difference $V_n=L-L_n={\displaystyle \sum _{t=n+1}^{\infty }x\left(t\right)}$ providing $V_n^2={\displaystyle \sum _{t=n+1}^{\infty }E\left\{x^2\left(t\right)|F_{t-1}\right\}}$, and $S_n^2=E\left\{V_n^2\right\}$. Let us assume further that the following hypotheses are satisfied:

(I)

The probability $P\left\{\left[{\displaystyle \frac{V_n^2}{S_n^2}}\right]>\epsilon \right\}<\delta$, for some $\epsilon >0$.

(II)

$\underset{n\to \infty }{\lim }{\displaystyle \frac{1}{S_n^2}}{\displaystyle \sum _{t=n+1}^{\infty }E\left\{x^2\left(t\right)I\left(\left|x\left(t\right)\right|\ge \epsilon S_n\right)\right\}}=0$, for all $\epsilon >0$, where $I\left(\cdot \right)$ is the indicator function. Then,

$${\displaystyle \frac{V_n}{S_n}}={\displaystyle \frac{L-L_n}{S_n}} \stackrel{D}{\to } N\left(0,1\right)$$

where the symbol $\stackrel{D}{\to }$ denotes the convergence in distribution, while $N\left(0,1\right)$ is the standard normal pdf with zero-mean and unit variance.

Lemma 3 

(Kronecker’s lemma) [45]. Let the sequence ${\displaystyle \sum _{i=1}^n{x}_i}$ of real numbers $x_i$ converge, and let $\left\{b_n\right\}$ be a monotonically increasing sequence of positive real numbers satisfying $\underset{n\to \infty }{\lim }{b}_n=\infty$ . Then,

$$\underset{n\to \infty }{\lim }{\displaystyle \frac{1}{b_n}}{\displaystyle \sum _{k=1}^n{b}_k{x}_k}=0$$

Lemma 4 

(Neveu’s martingale convergence lemma) [43]. Let $\left\{z_i\right\}$ be a sequence of non-negative real scalar random variables, while $\left\{F_i\right\}$ is a sequence of increasing sub- $\sigma$ -fields, defined on the probability space $\left(\Omega ,\mathrm{F},P\right)$ . Assume further that

$$E\left\{z_i|{\mathrm{F}}_{i-1}\right\}\le z_{i-1}+{\alpha }_i; {\displaystyle \sum _{i=1}^{\infty }{\alpha }_i}<\infty w.p.1$$

Then, the sequence $\left\{z_i\right\}$ converges w.p.1 to a non-negative finite random variable $z^{*}$ as the time index $i$ tends to infinity,

$$\underset{i\to \infty }{\lim }{z}_i=z^{*} w.p.1 or P\left\{\underset{i\to \infty }{\lim }{z}_i=z^{*}\right\}=1$$

Proceeding from the above lemmas, one can formulate the following Theorem 2, related to the asymptotic estimation error’s convergence to a zero-mean multidimensional normal pdf with the positive definite covariance matrix.

Theorem 2. 

Let the algorithm in (14)–(16) and model (4) satisfy the following hypotheses:

H1. 

The polynomial $A\left(\cdot \right)$ in (2) is a stable one.

H2. 

The sequence $\left\{e\left(i\right)\right\}$ in (4) consists of i.i.d. random variables, having a symmetric pdf $p,$ with zero mean $E_p\left\{e\left(i\right)\right\}=0$ and finite variance $E_p\left\{e^2\left(i\right)\right\}={\sigma }^2<\infty$.

H3. 

The observation vector $Z\left(i\right)$ in (5) is bounded, and $E\left\{Z\left(i\right)Z^T\left(i\right)\right\}$ is a positively definite matrix, i.e., $E\left\{Z\left(i\right)Z^T\left(i\right)\right\}>0$ , with $E\left\{\cdot \right\}$ being the expectation.

H4. 

The nonlinearity $\psi \left(\cdot \right)$ in (14) is an odd and bounded real scalar function.

H5. 

The first ${\psi }^{\prime }$ and second ${\psi }^{″}$ derivatives of the $\psi$ -function are bounded functions that exist for almost all real scalar arguments.

H6. 

The parameter estimate $\widehat{\theta }\left(i\right)$ in (14)–(16) exhibits strong consistency $P\left\{\underset{i\to \infty }{\lim }\widehat{\theta }\left(i\right)={\theta }_{*}\right\}=1$, with ${\theta }_{*}$ being the true parameter vector to be estimated.

Then, the normalized estimation error $\sqrt{i}\tilde{\theta }\left(i\right)$ converges asymptotically to a zero-mean multidimensional normal pdf,

$$\begin{array}{l}\sqrt{i}\tilde{\theta }\left(i\right)\stackrel{D}{ \to } N\left(0,V\right); \tilde{\theta }\left(i\right)=\widehat{\theta }\left(i\right)-{\theta }_{*}\\ V=\rho \left(\psi ,p\right){\left[E\left\{Z\left(i\right)Z^T\left(i\right)\right\}\right]}^{-1}; \rho \left(\psi ,p\right)={\displaystyle \frac{\beta \left(\psi ,p\right)}{{\alpha }^2\left(\psi ,p\right)}}={\displaystyle \frac{E_p\left\{{\psi }^2\left(\cdot \right)\right\}}{{E_p}^2\left\{{\psi }^{\prime }\left(\cdot \right)\right\}}}\end{array}$$

(17)

where $N\left(0,V\right)$ is the zero-mean multidimensional normal pdf, with the positive definite asymptotic covariance matrix $V$.

Following the methodology of Tsypkin, one can define the information matrix $D\left({\theta }_{*},{\sigma }^2\right)=E\left\{Z\left(i\right)Z^T\left(i\right)\right\}$, where ${\sigma }^2$ is the variance in H2, and, after substituting the observation vector $Z\left(i\right)$ in (5), the information matrix $D$ can be rewritten in the expanded form

$$D\left({\theta }_{*},{\sigma }^2\right)=E\left\{Z\left(i\right)Z^T\left(i\right)\right\}=D_1\left({\theta }_{*}\right)+{\sigma }^2{D}_2\left({\theta }_{*}\right)$$

(18)

Here, $D_1$ and $D_2$ are positive definite symmetric matrices [23,24,25,38].

Furthermore, let us note that the expression $\rho \left(\psi ,p\right)$ in (17) represents the asymptotic variance of Huber’s M-robust estimate of the scalar location parameter defined in (5), with $Z\left(i\right)=1$ [35]. Moreover, under the standard normal observation noise pdf $p\left(\cdot \right)\sim N\left(\cdot |0,1\right)$ with zero mean and unit variance, one gets from (9) the quadratic score function $F_0\left(z\right)=z^2/2$ with the first derivative ${\psi }_0\left(z\right)=z$ and the second one ${{\psi }^{\prime }}_0\left(z\right)=1$, yielding $\rho \left({\psi }_0,p\right)=1$ in (17). Thus, the algorithm in (14)–(16) reduces to the RLS one, with the asymptotic estimation error covariance $V_{LS}\left({\theta }_{*}\right)=E^{-1}\left\{Z\left(i\right)Z^T\left(i\right)\right\}$. Again, starting from (9), (17) and (18), one obtains, for a zero-mean normal noise pdf with the variance ${\sigma }^2,$ $p\left(\cdot \right)\sim N\left(\cdot |0,{\sigma }^2\right)$, that $\rho \left({\psi }_0,p\right)={\sigma }^2$, which further yields the asymptotic estimation error covariance matrix of the optimal RLS algorithm:

$$V\left({\widehat{\theta }}_{LS}\right)={\sigma }^2{V}_{LS}\left({\theta }_{*}\right); V_{LS}\left({\theta }_{*}\right)=E^{-1}\left\{Z\left(i\right)Z^T\left(i\right)\right\}$$

(19)

Usually, when one uses the RLS algorithm, the value of the noise variance ${\sigma }^2$ is not known. However, under certain conditions, it is possible to obtain a noise variance estimate as a byproduct of the parameter vector estimation [14,15,16,17].

The proof of Theorem 2 is presented in Appendix A. The proof is based on the standard martingale convergence results, combined with the given auxiliary lemmas. In addition, the Cramér–Rao theorem can be applied to the developed expression for the asymptotic covariance matrix of the estimation error $V$ in (17) to design an asymptotically efficient parameter estimation algorithm, (14)–(16), in the ML-based optimal framework under the exactly known observation noise pdf [1,2,9,23,24,25,38]. Moreover, starting from Huber’s min–max robust estimation approach, the concept of efficient parameter estimates may be extended to the case of observation noise uncertainty, resulting in the AML-based optimal recursive parameter estimation algorithm (14)–(16) on a specified class of pdfs [15,23,24,25,35,38]. The designs of the ML-based optimal and the AML-based optimal on class parameter estimation algorithms are presented in the next section.

5. Optimality and Robustness in Parameter Estimation

5.1. ML-Based Optimal Estimation Under Known Noise Statistics

Starting from the CR theorem and the developed asymptotic estimation error covariance matrix $V$ in (17), one can choose the score function $F\left(\cdot \right)$ in (10), as well as the measurement residual transformation $\psi \left(\cdot \right)$ in (14), optimally in the sense of achieving the maximum asymptotic convergence rate of the parameter estimates [1,2,9,23,24,25,38]. Thus, expression (17) indicates that the asymptotic estimation error covariance matrix $V$ depends on the real-scalar nonlinearity $\psi \left(\cdot \right)$ only through the scalar factor

$$\rho \left(\psi ,p\right)={\displaystyle \frac{\beta \left(\psi ,p\right)}{{\alpha }^2\left(\psi ,p\right)}}={\displaystyle \frac{E_p\left\{{\psi }^2\left(\cdot \right)\right\}}{E_p^2\left\{{\psi }^{\prime }\left(\cdot \right)\right\}}}={\displaystyle \frac{{\displaystyle {\displaystyle \underset{-\infty }{\overset{\infty }{\int }}}{\psi }^2\left(z\right)}p\left(z\right)dz}{{\left[{\displaystyle {\displaystyle \underset{-\infty }{\overset{\infty }{\int }}}{\psi }^{\prime }\left(z\right)}p\left(z\right)dz\right]}^2}}$$

(20)

where $p\left(\cdot \right)$ is the zero-mean pdf of i.i.d. observation noise samples $e\left(i\right)$ in (4). Therefore, the minimization of the matrix measure of estimation quality $V$ in (17) with respect to the scalar term $\psi$ reduces to minimizing the scalar deterministic factor $\rho$ in (20). The solution of the underlying variational problem is classical [23,24,25]: the optimal nonlinearity minimizing (20) under an exactly known pdf $p$ coincides, up to a multiplicative and an additive constant, with the ML-based optimal pair (9), i.e., ${\psi }_0=-{\left(\log p\right)}^{\prime }$.

In this context, the optimal recursive parameter estimation algorithm (14)–(16) utilizing the criterion (10) with the ML-based optimal score function $F_0$ in (9) produces the smallest possible asymptotic estimation error covariance matrix $V$ in (17) under the fully known observation noise pdf $p$. Using the ML-based optimal nonlinearity ${\psi }_0$ in (9) as the $\psi$-function in (14), one gets from (20) that the minimum scalar performance index $\rho \left({\psi }_0,p\right)$ in (20) is defined by

$$\begin{array}{c}\alpha \left({\psi }_0,p\right)=\beta \left({\psi }_0,p\right)=I\left(p\right);\\ \rho \left({\psi }_0,p\right)=I^{-1}\left(p\right); I\left(p\right)=E_p\left\{{\left(p^{\prime }/p\right)}^2\right\}={\displaystyle {\displaystyle \underset{-\infty }{\overset{\infty }{\int }}}\frac{{p^{\prime }}^2}{p}dp}\end{array}$$

(21)

Here, $I\left(p\right)$ denotes the scalar Fisher information [9,10]. Moreover, the scalar deterministic performance index $\rho \left({\psi }_0,p\right)$ satisfies the scalar optimality condition

$$\rho \left(\psi ,p\right)\ge \rho \left({\psi }_0,p\right); \psi \left(-z\right)=-\psi \left(z\right);p\left(-z\right)=p\left(z\right)$$

(22)

for any odd nonlinearity $\psi$ and any zero-mean symmetric noise pdf $p$.

The choice of the ML-based optimal nonlinearity ${\psi }_0$ in (9) for the nonlinear residual transformation $\psi$ in (14) is based on the Cramér–Rao approach [1,2,9,23,24,25,38]. In this sense, if the parameter estimates $\widehat{\theta }\left(i\right)$ in (14) are unbiased, $E\left\{\widehat{\theta }\left(i\right)\right\}={\theta }_{*}$, and if the components of the observation noise sequence $\left\{e\left(i\right)\right\}$ in (4) are i.i.d. random variables with a zero-mean symmetric pdf $p$, then the covariance matrix of the asymptotic estimation error $V$ in (17) satisfies the Cramér–Rao (CR) inequality,

$$V\left(\psi ,p\right)=\underset{i\to \infty }{\lim } E\left\{i\left[\widehat{\theta }\left(i\right)-{\theta }_{*}\right]{\left[\widehat{\theta }\left(i\right)-{\theta }_{*}\right]}^T\right\}\ge {\left[I\left(p\right)D\left({\theta }_{*},{\sigma }^2\left(p\right)\right)\right]}^{-1}$$

(23)

Here, ${\theta }_{*}$ is the true parameter vector in (4) to be estimated, while the observation or information matrix $D$ is defined by (18). The right-hand side of (23) is called the CR lower bound, and a parameter estimator that reaches the CR lower bound is called efficient, producing the maximum asymptotic convergence rate of the parameter estimates. Thus, if one adopts the ML-based optimal nonlinearity ${\psi }_0$ in (9) as the $\psi$-function in (14), the algorithm in (14)–(16) achieves the CR lower bound in (23), which presents the minimum asymptotic estimation error covariance matrix. Moreover, taking into account (17), (18) and (23), one obtains the optimality conditions

$$V\left(\psi ,p\right)\ge V\left({\psi }_0,p\right)=V_{CR}\left({\theta }_{*},p\right); V_{CR}\left({\theta }_{*},p\right)={\left[I\left(p\right)D\left({\theta }_{*},{\sigma }^2\left(p\right)\right)\right]}^{-1}$$

(24)

with $V_{CR}$ being the CR lower bound. Here, the matrix inequality denotes the non-negative definiteness of the matrix difference [31]. However, the actual zero-mean observation noise pdf $p$ with variance ${\sigma }^2\left(p\right)$ is rarely known exactly in practice.

5.2. Minimax-Robust Optimal ML Estimation over a Class of Noise Statistics

The problem of system parameter estimation under observation noise uncertainty can be considered a decision-making problem based on game theory with two players, in which an engineer and Nature have contradictory goals [23,24,25,38]. The goal of the engineer is to choose the nonlinearity $\psi$ in (14) so as to achieve the highest possible asymptotic convergence rate of the parameter estimates. In contrast, the goal of Nature is to adapt the parameter estimation procedure to the changing conditions of a local stochastic environment. Analogously to Huber’s min–max robust estimation approach, these requirements may be fulfilled by optimizing the parameter estimation algorithm under the worst-case situation within a local stochastic vicinity. This worst-case situation may be represented by the worst-case, or least favorable, pdf within a specified class $\mathrm{P}$ to which the real unknown observation noise pdf $p$ belongs [35]. Thus, starting from (17) and (23), one can define the min–max optimal matrix policy

$$\underset{\psi }{\min }\left\{\underset{p\in \mathrm{P}}{\max }V\left(\psi ,p\right)\right\}=V\left({\psi }_{*},p_{*}\right)=\underset{p\in \mathrm{P}}{\max }\left\{\underset{\psi }{\min }V\left(\psi ,p\right)\right\};V\left({\psi }_{*},p_{*}\right)=V_{CR}\left({\theta }_{*},p_{*}\right)$$

(25)

where the CR lower bound $V_{CR}$ is defined by (24). Furthermore, one obtains from (25) that the game has the saddle point pair $\left({\psi }_{*},p_{*}\right)$, where the function ${\psi }_{*}$ is referred to as the ML-based nonlinearity that is optimal on the class $\mathrm{P}$, while the pdf $p_{*}$ is referred to as the worst-case, or least favorable, pdf within the class $\mathrm{P}$ [23,24,25,35,38]. In accordance with the CR inequality (23), the choice of the optimal nonlinearity on a class ${\psi }_{*}$ as the nonlinearity $\psi$ in (14), as well as the optimal on a class score function $F_{*}$ for the score function $F$ in (10), determines the ML-based optimal on a class principle

$$F\left(z\right)=F_{*}\left(z\right)=-\log p_{*}\left(z\right); \psi \left(z\right)={\psi }_{*}\left(z\right)={\displaystyle \frac{d{F}_{*}\left(z\right)}{dz}}={F^{\prime }}_{*}\left(z\right)$$

(26)

This, in turn, leads to the ML-based parameter identification algorithm that is optimal on a class, defined by (14)–(16) and (26), where the worst-case pdf $p_{*}$ yields the maximum CR lower bound, in (26), within the specified class $\mathrm{P}$. Taking into account (24), one gets

$$p_{*}=\arg \left\{\underset{p\in \mathrm{P}}{\max }{V}_{CR}\left({\theta }_{*},p\right)\right\}==\arg \left\{\underset{p\in \mathrm{P}}{\min }I\left(p\right)D\left({\theta }_{*},{\sigma }^2\left(p\right)\right)\right\}$$

(27)

In this sense, the pair $\left({\psi }_{*},p_{*}\right)$ satisfies the saddle point conditions for the asymptotic error covariance matrix $V$ in (17) in the form

$$V\left(\psi ,p_{*}\right)\ge V\left({\psi }_{*},p_{*}\right)=V_{CR}\left({\theta }_{*},p_{*}\right)\ge V\left({\psi }_{*},p\right)$$

(28)

for any odd $\psi$-function in (14) and any zero-mean symmetric pdf $p$ from the adopted class $\mathrm{P}$. However, if there is no solution to the matrix optimization problem (27), the worst-case pdf $p_{*}$ does not exist, and the ML-based algorithm that is optimal on a class, given by (14)–(16), (26) and (27), is not realizable in practice. Furthermore, if an optimal solution $p_{*}$ in (27) does exist, it may also be derived by minimizing the scalar criterion $Tr\left\{I\left(p\right)D\left({\theta }_{*},{\sigma }^2\right)\right\}$, where $Tr\left\{\cdot \right\}$ denotes the matrix trace operation [23,24,25].

A solution to the nonclassical variational problem in (27) can be found in at least two scenarios: (i) a static system model, when the observation vector $Z\left(i\right)$ in (5) is constant; (ii) a linear dynamic system with a finite impulse response (FIR), where the observation vector $Z\left(i\right)$ in (5) contains only the samples of input signals that are uncorrelated with the white observation noise, reducing the matrix optimization problem in (27) to the condition of the minimum of the scalar Fisher information in (21). Furthermore, in the case of a linear dynamic system with an infinite impulse response (IIR), represented by the ARX form in (2), the solution for the optimal worst-case pdf $p_{*}$ in (27) can only be found numerically under the assumption that $\mathrm{P}$ is a class of pdfs with finite variance [23,24,25]. Moreover, if the unknown noise variance ${\sigma }^2\left(p\right)$ in (18) is somehow estimated from the observations, the observation matrix $D$ in (28) is no longer undetermined, and the matrix optimization problem in (27) is recast as the minimization of the scalar Fisher information in (21) within the specified class $\mathrm{P}$. A popular robust variance estimation procedure, utilizing at each stage a sliding data frame of suitable size, is the median absolute deviation (MAD) [32,33,34,35,36].

5.3. Huber M-Robustified AML Estimation over a Class of Noise Statistics

One can analyze the parameter identification problem using a slightly different approach. Thus, bearing in mind that the gain matrix $\Gamma \left(i\right)$ in (16) is an approximation, for a large enough $i$, of the asymptotic optimal gain matrix ${\Gamma }_0\left(i\right)={\left(iR\right)}^{-1}$, where the matrix $R$ is defined by the relation (A29) in Appendix A, one obtains that the current increment of the parameter vector estimates in (14), $\Delta \widehat{\theta }\left(i\right)=\widehat{\theta }\left(i\right)-\widehat{\theta }\left(i-1\right)$, conditioned by the previous input–output observations $\left\{Z\left(j\right),j\le i-1\right\}$, is given by

$$E\left\{i\Delta \widehat{\theta }\left(i\right){\Delta }^T\widehat{\theta }\left(i\right)\left|{\mathrm{F}}_{i-1}\right.\right\}=i^{-1}\rho \left(\psi ,p\right)E^{-1}\left\{Z\left(i\right)Z^T\left(i\right)|{\mathrm{F}}_{i-1}\right\}$$

(29)

where the scalar performance index $\rho \left(\cdot \right)$ is defined by (20). In addition, by replacing the preceding estimate $\widehat{\theta }\left(i-1\right)$ with the expectation $E\left\{\widehat{\theta }\left(i-1\right)\right\}={\theta }_{*}$, with ${\theta }_{*}$ being the true parameter vector in (4), one obtains further the conditional covariance matrix of the estimation error $\tilde{\theta }\left(i\right)=\widehat{\theta }\left(i\right)-{\theta }_{*}$ at the present i-th stage:

$$E\left\{i\tilde{\theta }\left(i\right){\tilde{\theta }}^T\left(i\right)\left|{\mathrm{F}}_{i-1}\right.\right\}=i^{-1}\rho \left(\psi ,p\right)E^{-1}\left\{Z\left(i\right)Z^T\left(i\right)|{\mathrm{F}}_{i-1}\right\}; \tilde{\theta }\left(i\right)=\widehat{\theta }\left(i\right)-{\theta }_{*}$$

(30)

Furthermore, since the information vector $Z\left(i\right)$ in (5) is known at the current stage $i$, the min–max optimality policy, related to the conditional estimation error covariance in (30), takes the form

$$\underset{\psi }{\min }\left\{\underset{p\in \mathrm{P}}{\max }\rho \left(\psi ,p\right)\right\}=\rho \left({\psi }_{*},p_{*}\right)=\underset{p\in \mathrm{P}}{\max }\left\{\underset{\psi }{\min }\rho \left(\psi ,p\right)\right\}$$

(31)

for any odd $\psi$-function and any pdf $p$ from the specified class. Taking into account (9), (21) and (22), where the pair $\left({\psi }_0,p\right)$ in (9) is replaced by the pair $\left({\psi }_{*},p_{*}\right)$ in (31), the AML-based optimal on a class nonlinearity is defined by

$${\psi }_{*}\left(\cdot \right)=-{\left[\log p_{*}\left(\cdot \right)\right]}^{ \prime }; p_{*}\left(\cdot \right)=\arg \underset{p\in \mathrm{P}}{\min }I\left(p\right)$$

(32)

Here, the suboptimal worst-case pdf $p_{*}\left(\cdot \right)$ in (32) approximates the optimal worst-case pdf $p_{*}\left(\cdot \right)$ in (27) assuming that the unknown noise variance ${\sigma }^2\left(p\right)$ in (18) is somehow estimated. Thus, starting from (21) and (22), one concludes that the pair $\left({\psi }_{*},p_{*}\right)$ in (32) satisfies the saddle point conditions for the scalar performance index $\rho$ in (20)—that is,

$$\rho \left(\psi ,p_{*}\right)\ge \rho \left({\psi }_{*},p_{*}\right)\ge \rho \left({\psi }_{*},p\right); \rho \left({\psi }_{*},p_{*}\right)=I^{-1}\left(p_{*}\right)$$

(33)

In this sense, an AML-based parameter estimation algorithm that is optimal on a class, given by (14)–(16) and (32), is step-by-step optimal, achieving, in each step, the minimum conditional estimation error covariance matrix in (30), under the suboptimal worst-case pdf $p_{*}$ in (32). The latter ensures the minimum Fisher information $I\left(p_{*}\right)$ in (33), limiting, at each stage, the conditional covariance matrix of the estimation error in (30) for any pdf $p$ within the specified class $\mathrm{P}$. This, in turn, desensitizes the parameter estimation algorithm (14)–(16) and (32) to outliers at each stage, making the algorithm robust in the practical sense.

The AML-based algorithm that is optimal on a class, given by (14)–(16) and (32), is determined by the approximate nonlinearity ${\psi }_{*}\left(\cdot \right)$, optimal on a class, given in (32), and the statistical linearization coefficient $\alpha =E_p\left\{{{\psi }^{\prime }}_{*}\left(\cdot \right)\right\}$ in (15). The latter cannot be computed exactly, since the real observation noise pdf $p$ is not exactly known. Therefore, one has to apply some realizable approximation of the coefficient $\alpha$. One possibility is to calculate the expectation $\alpha$ in (15) with respect to the suboptimal worst-case pdf $p_{*}$ in (32), resulting in the fixed factor denoted by ${\alpha }_{*}.$ Another possibility is to apply the current residual realization to approximate the unknown expectation $\alpha$, resulting in the variable factor denoted by ${\alpha }_k$ at the current stage $k$. Thus, starting from (15), (21), (31) and (32), one gets

$$\begin{array}{c}\alpha =E_p\left\{{{\psi }_{*}}^{\prime }\left(\cdot \right)\right\}; {\alpha }_{*}=E_{p_{*}}\left\{{{\psi }_{*}}^{\prime }\left(\cdot \right)\right\}=I\left(p_{*}\right); {\alpha }_k={{\psi }_{*}}^{\prime }\left[\epsilon \left(k\right)\right];\\ {{\psi }_{*}}^{\prime }\left(z\right)\approx {\displaystyle \frac{\psi \left(z\right)}{z}}, {{\psi }_{*}}^{\prime }\left(0\right)=1\end{array}$$

(34)

with $\epsilon \left(\cdot \right)$ being the measurement residual in (15). The practical implementation aspects of the proposed recursive, M-robustified AML algorithm, optimal on a class, for parameter estimation are discussed in the following.

6. Implementation and Statistical Validation

6.1. Implementation

The real-time application of the AML-based recursive parameter estimation algorithm that is optimal on a class, under observation noise uncertainty, given by (14)–(16), (32) and (34), requires the definition of a class $\mathrm{P}$ to which the real unknown observation noise pdf belongs, as well as finding the suboptimal worst-case pdf $p_{*}$ in (32). Some examples of typical classes are presented in the literature [23,24,25,35,38].

In accordance with the central limit theorem of mathematical statistics, it is commonly assumed in various engineering and scientific problems that real observations exhibit a distribution that closely approximates a zero-mean Gaussian one, with finite variance ${\sigma }^2$ [9,10,11,12,13,14,15,16,17]. In this sense, a natural choice is a class $\mathrm{P}$ of observation noise pdfs $p$ in the form

$$\mathrm{P}=\left\{p|{\displaystyle {\displaystyle \underset{-\infty }{\overset{\infty }{\int }}}{z}^{2}p\left(z\right)dz}<{\sigma }^2<\infty \right\}$$

(35)

Thus, the AML-based optimal pair $\left(p_{*},{\psi }_{*}\right)$ in (32) on the class $\mathrm{P}$ in (35), as well the fixed-factor approximation ${\alpha }_{*},$ of the linearization coefficient $\alpha$ in (34), is defined by [23,24,25,32,33,34,35,36,38]

$$p_{*}\left(z\right)=N\left(z|0,{\sigma }^2\right)={\displaystyle \frac{1}{\sqrt{2\pi }\sigma }}{e}^{-{\displaystyle \frac{z^2}{2{\sigma }^2}}}; {\psi }_{*}\left(z\right)={\displaystyle \frac{z}{{\sigma }^2}}; \alpha \approx {\alpha }_{*}=I\left(p_{*}\right)={\sigma }^{-2}$$

(36)

By substituting (36) into (14)–(16), and replacing again $\Gamma \left(i\right)/{\sigma }^2$ with $\Gamma \left(i\right)$, one gets the RLS algorithm

$$\widehat{\theta }\left(i\right)=\widehat{\theta }\left(i-1\right)+\Gamma \left(i\right)Z\left(i\right)\epsilon \left(i,\widehat{\theta }\left(i-1\right)\right); \widehat{\theta }\left(0\right)=0$$

(37)

$$\epsilon \left(i,\widehat{\theta }\left(i-1\right)\right)=y\left(i\right)-Z^T\left(i\right)\widehat{\theta }\left(i-1\right)$$

(38)

$$\Gamma \left(i\right)=\Gamma \left(i-1\right)-{\displaystyle \frac{\Gamma \left(i-1\right)Z\left(i\right)Z^T\left(i\right)\Gamma \left(i-1\right)}{1+Z^T\left(i\right)\Gamma \left(i-1\right)Z\left(i\right)}}; \Gamma \left(0\right)=\gamma I$$

(39)

with $I$ being an identity matrix, while $\gamma$ is a positive constant. However, the LS-type identification algorithm, denoted by ${\widehat{\theta }}_{LS}$, exhibits a high degree of sensitivity to small deflections of the real observation noise pdf $p$ from the normal one, thus being non-robust and possibly even failing to converge. In this sense, real measurements typically include unavoidable multiple outliers, ranging from five to ten percent within predominantly Gaussian-distributed data. Therefore, the class of approximately normal pdfs corrupted by outliers, the so-called $\delta$-contaminated Gaussian pdfs

$$\mathrm{P}=\left\{p\left(z\right)|p\left(z\right)=\left(1-\delta \right)p_n\left(z\right)+\delta p_0\left(z\right); 0\le \delta <1\right\},$$

(40)

represents, in practice, the most widespread form of available prior information regarding the statistical properties of the observation noise $e\left(i\right)$ in (4) [23,24,25,32,33,34,35,36,38]. Here, $p_n\left(\cdot \right)$ in (40) is the zero-mean standard normal pdf $N\left(\cdot |0,{\sigma }_n^2\right)$ with unit variance ${\sigma }_n^2=1$. The parameter $\delta$ is called the contamination degree, describing the probability of outlying data samples arising from pdf $p_0\left(\cdot \right)$ in (40), which has large variance ${\sigma }_0^2\gg {\sigma }_n^2$. The worst-case pdf $p_{*}$ in (32), minimizing the Fisher information $I\left(p\right)$ in (21) within the given pdf class $\mathrm{P}$ in (40), is the heavy-tailed Gaussian one, with the tails belonging to the Laplacian pdf

$$p_{*}\left(z\right)=\arg \underset{p\in \mathrm{P}}{\min }I\left(p\right)=\left\{\begin{array}{l}{\displaystyle \frac{1-\epsilon }{\sqrt{2\pi }{\sigma }_n}}\exp \left(-{\displaystyle \frac{z^2}{2{\sigma }_n^2}}\right); \left|z\right|\le \Delta \hfill \\ {\displaystyle \frac{1-\epsilon }{\sqrt{2\pi }{\sigma }_n}}\exp \left({\displaystyle \frac{{\Delta }^2}{2{\sigma }_n^2}}-{\displaystyle \frac{\Delta \left|z\right|}{{\sigma }_n^2}}\right); \left|z\right|>\Delta \hfill \end{array}\right.$$

(41)

As mentioned before, the nominal noise variance ${\sigma }_n^2$ in (41) may be estimated from the measurement data by using the robust MAD estimator [23,24,25,32,33,34,35,36,38]. Starting from (41), the AML-based optimal on a class score function $F_{*}\left(\cdot \right)$ in (26) and the associated AML-based optimal nonlinear residual, or prediction error, transformation ${\psi }_{*}\left(\cdot \right)$ in (32) are represented by

$$F_{*}\left(z\right)=-\log p_{*}\left(z\right)=\left\{\begin{array}{l}{\displaystyle \frac{\sqrt{2\pi }\sigma }{1-\delta }}+{\displaystyle \frac{z^2}{2{\sigma }_n^2}};\left|z\right|\le \Delta \hfill \\ {\displaystyle \frac{\sqrt{2\pi }{\sigma }_n}{1-\delta }}-{\displaystyle \frac{{\Delta }^2}{2{\sigma }_n^2}}+{\displaystyle \frac{\left|z\right|\Delta }{{\sigma }_n^2}};\left|z\right|>\Delta \hfill \end{array}\right.$$

(42)

$${\psi }_{*}\left(z\right)=-{\left[\log p_{*}\left(z\right)\right]}^{ \prime }=\min \left\{{\displaystyle \frac{\left|z\right|}{{\sigma }_n^2}},{\displaystyle \frac{\Delta }{{\sigma }_n^2}}sgn\left(z\right)\right\}; \Delta =k{\sigma }_n; k=k\left(\delta \right)$$

(43)

where $sgn$ denotes the sign function. Here, the dependence of the tuning parameter $\Delta$ on the contamination degree $\delta$ follows from the normalizing condition ${\displaystyle {\displaystyle \underset{-\infty }{\overset{\infty }{\int }}}{p}_{*}\left(z\right)dz}=1$, yielding

$${\displaystyle {\displaystyle \underset{-\Delta }{\overset{\Delta }{\int }}}N\left(z|0,{\sigma }_n^2\right)dz }+{\displaystyle \frac{2{\sigma }_n}{\Delta }}N\left(\Delta |0,{\sigma }_n^2\right)={\left(1-\delta \right)}^{-1}$$

(44)

The integral in (44) can be solved numerically by utilizing the error function [9,10].

Function $erf\left(x\right)$ is calculated numerically for the set of arguments $\left\{x_i=i\Delta x\right.$, $i=1,\dots ,60$, $\left.\Delta x=0.05\right\}$ and stored in a statistical table. By introducing the normalized tuning parameter $k=\Delta /{\sigma }_n$ in (43), the relation (44) can be rewritten in the form

$$2erf\left(k\right)+{\displaystyle \frac{\sqrt{2/\pi }}{k}}{e}^{- {\displaystyle \frac{k^2}{2}}}={\displaystyle \frac{1}{1-\delta }}; k=\Delta /{\sigma }_n$$

(45)

In this context, one can calculate, for the chosen set of values $\left\{k_i=i\Delta k\right.$, $i=1,\dots ,60$, $\left.\Delta k=0.05\right\}$, the set of associated values $\left\{{\delta }_i, i=1,\dots ,60\right\}$ by utilizing (45) and the stored values of the $erf$-function, combined with the linear interpolation technique. The obtained set of pairs $\left\{\left(k_i,{\delta }_i\right)\right.$, $\left.i=1,\dots ,60\right\}$ can be used further to evaluate the values of the saturation threshold $k=k\left(\delta \right)$ in (43) for desired values of the contamination degree $\delta$ in (40). The calculated values of the normalized tuning parameter $k\left(\delta \right)$ in (45) for different values of the parameter $\delta$ in (40) are presented in

Table 1. Dependence of tuning parameter $k\left(\delta \right)$ in (43) on $\delta$-contamination degree in (40).

$\delta$	0.00	0.01	0.02	0.05	0.10	0.20	0.50	1.00
$k\left(\delta \right)$	$\infty$	2.00	1.70	1.40	1.10	0.90	0.40	0.00

.

It should be noted that, even for small values of the parameter $\delta$, the associated values of the tunning parameter $k\left(\delta \right)$ are rather small. Moreover, the underlying fixed factor ${\alpha }_{*}$ in (34) corresponding to the least-favorable pdf $p_{*}$ in (41), as well as the scalar performance index $\rho \left({\psi }_{*},p_{*}\right)$ in (33), is given by

$${\alpha }_{*}=I\left(p_{*}\right)={\displaystyle {\displaystyle \underset{-\Delta }{\overset{\Delta }{\int }}}{{\psi }^{\prime }}_{*}\left(z\right)p_{*}\left(z\right)dz}=2\left(1-\delta \right)erf\left(k\right), \Delta =k{\sigma }_n; \rho \left({\psi }_{*},p_{*}\right)=I^{-1}\left(p_{*}\right)$$

(46)

In this context, the tuning parameter $k$ in (46) can be chosen to give a required efficiency at the nominal Gaussian pdf $p_n$ in (40). Furthermore, since the $erf\left(k\right)$ function asymptotically approaches the value $0.5$, the resulting fixed-factor value ${\alpha }_{*}$ asymptotically represents the probability $1-\delta$ of the regular measurements arising from $p_n$ in (40). Thus, the contamination degree $\delta$ represents the probability of outlier occurrence. Unfortunately, the parameter $\delta$ is unknown beforehand, so it would need to be estimated. However, an estimation procedure based on the residual sequence $\left\{\epsilon \left(i\right)\right\}$ in (15) is not always successful [23,24,25,32,33,34,35,36,38]. For this reason, one has to adopt either the fixed value of the tuning parameter $k$ from Table 1 in advance or the variable factor ${\alpha }_k$ in (34), which does not require exact knowledge of the $\delta$-contamination degree. It should also be noted that hypothesis H5 of Theorem 2 is not satisfied for the nonlinearity ${\psi }_{*}$ in (43). However, this problem is overcome by generalizing the first derivative of the optimal ${\psi }_{*}$-nonlinearity using the suitable form (34). More generally, hypothesis H5 of Theorem 2 requires bounded first and second derivatives of the ψ-function everywhere, whereas the AML-based nonlinearity ψ* of (43) fails to satisfy this requirement exactly at the kink points defining the saturation region. In practice, this discrepancy is immaterial, since the generalized derivative given in (34) is used in place of the exact one when computing the statistical linearization coefficient, so that the asymptotic results of Theorem 2 remain applicable to the M-robustified algorithm even though the underlying nonlinearity is only piecewise, rather than everywhere, continuously differentiable.

The proposed AML-based identification algorithm, optimal on a class, is closely related to Huber’s M-robust parameter estimates in static regression plants. Thus, the AML-based nonlinearity ${\psi }_{*}$ in (43), optimal on a class, coincides with the Huber M-robust influence function [35].

6.2. Statistical Validation

Starting from the CR inequality in (23) and (24), one may analyze the practical robustness feature of the parameter estimation procedure, denoted by $\widehat{\theta }$, by utilizing the relative asymptotic estimation efficiency quantitative criterion ($RAEF$) [38]. This criterion is defined as the ratio between the determinant of the CR lower bound $V_{CR}\left({\theta }_{*},p\right)$ in (24) and the determinant of the covariance matrix of the asymptotic estimation error $V\left(\widehat{\theta }\right)$ of the parameter estimator $\widehat{\theta }$ in question—that is,

$$\begin{array}{c}RAEF\left(\widehat{\theta },p\right)={\displaystyle \frac{\det V_{CR}\left({\theta }_{*},p\right)}{\det V\left(\widehat{\theta }\right)}}; V_{CR}\left({\theta }_{*},p\right)=I^{-1}\left(p\right)V_{LS}\left({\theta }_{*}\right);\\ V_{LS}\left({\theta }_{*}\right)=E^{-1}\left\{Z\left(i\right)Z^T\left(i\right)\right\}\end{array}$$

(47)

where $V_{LS}\left({\theta }_{*}\right)$ is defined by (18). In addition, $p$ designates the real observation noise pdf, while ${\theta }_{*}$ is the true parameter vector in (4). In particular, for the proposed AML-based parameter estimation procedure that is optimal on a class, given by (14)–(16), (43) and (46) and denoted by ${\widehat{\theta }}_{\rho }$, the associated asymptotic estimation error covariance matrix $V\left({\widehat{\theta }}_{\rho }\right)$ is represented by the matrix $V\left({\psi }_{*},p_{*}\right)$. The latter is defined by (17), (33) and (46), yielding further from (47)

$$RAEF\left({\widehat{\theta }}_{\rho },p\right)={\displaystyle \frac{\det V_{CR}\left({\theta }_{*},p\right)}{\det V\left({\psi }_{*,}{p}_{*}\right)}}={\displaystyle \frac{I^{-1}\left(p\right)}{\rho \left({\psi }_{*},p_{*}\right)}}={\displaystyle \frac{I^{-1}\left(p\right)}{I^{-1}\left(p_{*}\right)}}$$

(48)

Here, $p_{*}$ is the suboptimal worst-case pdf in (41), yielding the minimum Fisher information $I\left(p_{*}\right)$ in (46) for the threshold value $k=1.5$.

In this context, by using the criterion (48), one may analyze the efficiency loss caused by the outliers corrupting the Gaussian-distributed measurements. Here, the pdf $p$ is the standard normal pdf $N\left(\cdot |0,1\right)$ with zero mean and unit variance, yielding the unit value of the Fisher information $I\left(p\right)=1$. The calculated values from (48) for different $\delta$-values in (40) are presented in

Table 2. Dependence of $RAEF$ criterion (48) on $\delta$-contamination degree in (40).

$\delta$	0.00	0.01	0.02	0.05	0.10	0.20	0.50	1.00
$RAEF$	0.87	0.86	0.85	0.82	0.78	0.74	0.69	0.43

. In this sense, the minimum Fisher information $I\left(p_{*}\right)$ in (48) within the class $\mathrm{P}$ in (40) is calculated from (46) for each $\delta$-value of interest.

The obtained results are compared with the unit-valued $RAEF$ criterion, corresponding to the linear optimal LS parameter estimator ${\widehat{\theta }}_{LS}$ under the standard normal pdf $N\left(\cdot |0,1\right)$. Thus, if the value of the criterion (48) is closer to one, the parameter estimator ${\widehat{\theta }}_{\rho }$ is asymptotically more efficient for the particular $\delta$-value in (40), indicating a smaller efficiency loss. The results presented in Table 2 also show that the nonlinear parameter estimator ${\widehat{\theta }}_{\rho }$ is not significantly inferior to the linear optimal one ${\widehat{\theta }}_{LS}$ under the standard Gaussian pdf $N\left(\cdot |0,1\right)$ corresponding to the value $\delta =0,$ making it robust in the efficiency robustness sense. Furthermore, for small and moderate values of the $\delta$-contamination degree in (40), belonging to the interval $\delta \in \left(0.0, 0.2\right)$, the efficiency loss of the estimation procedure ${\widehat{\theta }}_{\rho }$ is rather small, making the latter also robust in the resistance robustness sense. Thus, since the parameter estimator ${\widehat{\theta }}_{\rho }$ produces fairly good results under both purely Gaussian-distributed data and Gaussian data corrupted by outliers, it also achieves practical robustness, including efficiency and resistance robustness performance.

In this sense, an important practical question is related to the estimation quality of the nonlinear parameter estimator ${\widehat{\theta }}_{\rho }$ in situations when the actual observation noise pdf $p$ in (48) is not confined to the adopted class of pdfs $\mathrm{P}$ in (40). In this context, the performance index $RAEF$ in (48) is calculated for a number of observation noise pdfs $p$, some of which do not belong to the class $\mathrm{P}$ in (40). The obtained results are presented in

Table 3. RAEF criterion (48) for various observation noise statistics.

$\mathit{R}\mathit{A}\mathit{E}\mathit{F}$ Criterion in (48)
$\mathit{p}\mathit{d}\mathit{f}$	$\mathit{N}\left(0,2\right)$	$\mathit{R}\left(-2,2\right)$	$\mathit{L}\left(0,0.5\right)$	$\mathit{C}\left(0,1\right)$	$\mathit{\delta }=0.1, {\mathit{p}}_0=\mathit{N}\left(0,10\right)$ in (40)
${\sigma }^2$	2.00	1.33	2.0	$\infty$	10.9
$I\left(p\right)$	0.50	0.0625	0.25	0.50	0.80
$RAEF$	1.60	13.90	3.20	1.60	1.09
Algorithm: ${\widehat{\theta }}_{\rho }$: (14)–(16), (43) and (46), $k=1.5$, $\delta =0.1$
Description: $p$—noise pdf; ${\sigma }^2$—variance; $I\left(p\right)$—Fisher information
	$N\left(z|0,{\sigma }^2\right)=\left(1/\sqrt{2\pi }\sigma \right)\exp \left\{-z^2/2{\sigma }^2\right\}$ zero-mean normal pdf; $I\left(p\right)={\sigma }^{-2}$
	$R\left(a,b\right)$—uniform pdf on interval $\left(a,b\right)$; ${\sigma }^2={\left(b-a\right)}^2/12$; $I\left(p\right)=1/{\left(b-a\right)}^2$
	$L\left(z|0,\lambda \right)=\left(\lambda /2\right)\exp \left\{-\lambda \left|z\right|\right\}$ zero-mean Laplace; ${\sigma }^2=2/{\lambda }^2$, $I\left(p\right)={\lambda }^2$
	$C\left(0,\lambda \right)=\left(\lambda /\pi \right){\left({\lambda }^2+z^2\right)}^{-1}$ zero-mean Cauchy pdf, ${\sigma }^2=\infty$.

.

It should be noted that the values of the RAEF performance index in (48) are not less than one due to the boundedness feature of the ${\psi }_{*}$-nonlinearity in (43), which makes the denominator $I\left(p\right)\rho \left({\psi }_{*},p_{*}\right)$ in (48) less than one for a symmetric heavy-tailed pdf $p$. In this context, if the value of the criterion (48) is closer to one, the ${\widehat{\theta }}_{\rho }$-estimation procedure is more efficient under the particular pdf $p$. The presented results in Table 3 show that the nonlinear parameter estimator ${\widehat{\theta }}_{\rho }$ produces practically acceptable quality of estimation under a zero-mean symmetric heavy-tailed observation noise pdf $p$. Typical examples of such pdfs are the Gaussian mixture, Cauchy and Laplacian pdfs. Here, the Fisher information $I\left(p\right)$ depends on a single shape parameter of the pdf p of interest. The best results are obtained for the Gaussian mixture pdf, belonging to the class $\mathrm{P}$ in (40), while slightly inferior results are obtained under the Cauchy pdf, as well the normal pdf. Moreover, the Laplacian pdf exhibits exponential tail decay rather than the quadratic-exponential decay of the nominal Gaussian pdf $p_n$ in (40), meaning that large-amplitude samples are relatively more probable, corresponding to higher $\delta$-values than the assumed value $\delta =0.1$. Therefore, smaller $k$-values than the adopted value $k=1.5$ may produce better quality of estimation (see Table 1). From this point of view, the Laplacian pdf $L\left(0,\lambda \right)$ is the worst-case pdf $p_{*}$ in (32) within the class $\mathrm{P}$ of pdfs $p$, being continuous in the origin $p\left(0\right)\ge \lambda /2>0$ [15,23,24,25,38]. Thus, the optimal-on-a-class nonlinearity ${\psi }_{*}$ in (32) reduces to a saturation-type $\mathrm{sgn}$-function, yielding inferior results compared to those obtained for the previously mentioned pdfs. Finally, rather poor estimation quality is produced under the uniform noise pdf $\mathrm{R}\left(-a,a\right)$, generating equally likely data samples on the given interval of length $2a$. In this sense, a class of $\delta$-contaminated uniform pdfs may be obtained from (40) by replacing the nominal normal pdf $p_n$ with a uniform pdf $\mathrm{R}\left(-a,a\right)$. In accordance with (41), the worst-case pdf $p_{*}\left(z\right)$ in (32) on the given class is the uniform one in the middle for $\left|z\right|\le a$ and the Laplacian one in the tails for $\left|z\right|>a$. Thus, the nonlinearity ${\psi }_{*}$ in (32), optimal on this class, reduces to a saturation-type $\mathrm{sgn}$-function in the tails $\left|z\right|>a$ with a dead zone in the middle $\left|z\right|\le a$. This nonlinearity deviates significantly from the optimal nonlinearity ${\psi }_{*}$ in (43) under the class $\mathrm{P}$ in (40) [15,23,24,25,38]. Finally, as mentioned before, the nonlinear robust estimator ${\widehat{\theta }}_{\rho }$ is slightly inferior to the optimal linear estimator ${\widehat{\theta }}_{LS}$ under the pure standard Gaussian pdf $N\left(0,1\right)$ (see Table 2). This is the price paid for achieving practical robustness involving both the efficiency and resistance robustness features.

The theoretical analysis is mostly based on the asymptotic properties of the proposed AML-based optimal on a class-recursive identification algorithm ${\widehat{\theta }}_{\rho }$ assuming an infinite size of the measurement data. However, it is important to investigate the properties of estimator ${\widehat{\theta }}_{\rho }$ under measurement data of finite length, bearing in mind the limited number of observations in real-time applications. Therefore, an important engineering example of linear dynamic system parameter identification under Gaussian-distributed observations corrupted by impulsive noise, or outliers, is also analyzed using Monte Carlo simulations.

7. Simulation Example and Experimental Validation

The presented simulation results are obtained for the ARX model (2), with the underlying polynomials in (3), given by

$$A\left(q^{-1}\right)=1-1.5q^{-1}+0.7q^{-2}, B\left(q^{-1}\right)=q^{-1}+0.5q^{-2}$$

(49)

The white random input sequence $\left\{u\left(i\right)\right\}$ in (2) is generated by the standard normal pdf $N\left(\cdot |0,1\right),$ while the white observation noise sequence $e\left(i\right)$ in (2) is generated from the zero-mean Gaussian mixture pdf in (40), with ${\sigma }_n^2=1$ and $p_0\left(\cdot \right)~N\left(\cdot |0,{\sigma }_0^2\right)$, ${\sigma }_0^2\gg 1$. The noise sample $e\left(i\right)$ at the present i-th stage is generated by first producing the sample $r$ from a uniform or rectangular pdf $\mathrm{R}\left(0,1\right)$. Thus, if $r\le \delta$, with $\delta$ being the given contamination degree in (40), the noise sample $e\left(i\right),$ is generated from the contaminating zero-mean normal pdf $p_0\left(\cdot \right)=N\left(\cdot |0,{\sigma }_0^2\right)$ with the given large variance ${\sigma }_0^2\gg 1$; otherwise, the noise sample $e\left(i\right)$ is generated from the nominal standard normal pdf $p_n\left(\cdot \right)=N\left(\cdot |0,1\right)$ in (40).

Furthermore, the following algorithms have been tested: the linear RLS (37)–(39), optimal under the standard Gaussian pdf $N\left(0,1\right)$ and denoted by A1; the AML-based identification algorithm, optimal on the class in (40), defined by (14)–(16), (43) and (46) and designated as A2; and Algorithm A2, using the variable factor ${\alpha }_k$ in (34) instead of the fixed factor ${\alpha }_{*}$ in (46), denoted by A3 (see

Table 4. Flowchart of Algorithm A3 with variable linearization coefficient.

Step 0:  Set the initial values: $\widehat{\theta }\left(0\right)$ in (14); $\Gamma \left(0\right)$ in (16); threshold level $k$ in (41); nominal noise variance ${\sigma }_n^2$ in (40); polynomial orders $n$ and $m$ in (2), $n\ge m$; number of iterations $N_s$; initial regression vector in (3), $Z^T\left(n\right)=\left\{-y\left(n-1\right),\dots ,-y\left(0\right),u\left(n-1\right),\dots ,u\left(n-m\right)\right\}$;  Define influence function ${\psi }_{*}$ in (41) for the given $k$ value.

Step 1: Time-counter initialization, $i=n$; $\widehat{\theta }\left(n-1\right)=\widehat{\theta }\left(0\right)$; $\Gamma \left(n-1\right)=\Gamma \left(0\right)$ Step 2: Calculate the measurement residual in (14), $\epsilon \left(i\right)=y\left(i\right)-Z^T\left(i\right)\widehat{\theta }\left(i-1\right)$ Step 3: Calculate the residual nonlinear transformation $\psi \left(i\right)={\psi }_{*}\left(\epsilon \left(i\right)\right)$ Step 4: Calculate the variable linearization factor $\alpha$ in (34), $\alpha \left(i\right)=\left\{\begin{array}{l}{\displaystyle \frac{\psi \left(i\right)}{\epsilon \left(i\right)}} \mathrm{if} \epsilon \left(i\right)\ne 0\hfill \\ 1 \mathrm{if} \epsilon \left(i\right)=0\hfill \end{array}\right.$ Step 5: Calculate the gain matrix from (16), $\Gamma \left(i\right)=\Gamma \left(i-1\right)-{\displaystyle \frac{\Gamma \left(i-1\right)Z\left(i\right)Z^T\left(i\right)\Gamma \left(i-1\right)}{\alpha {\left(i\right)}^{-1}+Z^T\left(i\right)\Gamma \left(i-1\right)Z\left(i\right)}}$ Step 6: Calculate the parameter estimate update $\widehat{\theta }\left(i\right)=\widehat{\theta }\left(i-1\right)+\Gamma \left(i\right)Z\left(i\right)\psi \left(i\right)$ Step 7: Increase time counter $i\leftarrow i+1$ Step 8:   IF $i\le N_S$ THEN    Memorize the parameter estimates $\widehat{\theta }\left(i\right)$,    Take the current input–output signals $y\left(i\right), u\left(i\right)$,    Upgrade the regression vector $Z^T\left(i\right)=\left\{-y\left(i-1\right),\dots ,-y\left(i-n\right),u\left(i-1\right),\dots ,u\left(i-m\right)\right\}$,    where $y\left(j\right)=0,u\left(j\right)=0$ for $j<0$    GOTO Step 2   ENDIF End of the simulation

Note: The flowchart describes the nonlinear Algorithm A3. If the fixed linearization coefficient $\alpha$ in (46) is used instead of the variable coefficient, the flowchart reduces to Algorithm A2; if the fixed coefficient is chosen to be $\alpha =1$, and the linear transformation $\psi \left(x\right)=x$ is used instead of the nonlinear one in (41), the flowchart reduces to the RLS algorithm.

).

The fixed factor ${\alpha }_{*}$ in (46) approximates the expected slope of the nonlinearity ${\psi }_{*}$ in (43), optimal on the class in (40), based on the least favorable pdf $p_{*}$ in (41) instead of the real unknown noise pdf. However, as mentioned before, such an approximation requires the definition in advance of the contamination degree $\delta$ in (40). On the other hand, the variable factor ${\alpha }_k$ in (34) represents the current slope of the AML-based optimal nonlinearity ${\psi }_{*}$ in (43), so that knowledge of the $\delta$-contamination degree is not required (see Section 5).

The effects of making the parameter estimates insensitive to outliers, contaminating the mainly Gaussian-distributed data, are presented in Figure 1, depicting the normalized mean square error, $NMSE$, defined by

$$NMSE\left(k\right)={\displaystyle \frac{1}{M}}{\displaystyle \sum _{j=1}^M\frac{{‖{\widehat{\theta }}_j\left(k\right)-{\theta }_{*}‖}^2}{{‖{\theta }_{*}‖}^2}}; k=1,\dots ,N; NMS={\displaystyle \frac{1}{N}}{\displaystyle \sum _{k=1}^{N}NMSE\left(k\right)}$$

(50)

where $NMS$ represents the average performance measure.

The $NMSE$ estimation quality measure is calculated by using $M=100$ Monte Carlo runs and $N=2000$ steps. Here, $‖\cdot ‖$ is the Euclidean norm, while ${\theta }_j\left(k\right)$ is the parameter vector estimate at the k-th stage for the j-th Monte Carlo run, and ${\theta }_{*}$ is the true parameter vector in (49).

Figure 2 depicts the typical behavior of the analyzed algorithms under a single Monte Carlo trial ($M=1$).

The effects of desensitizing the parameter estimates with respect to outlier statistics, involving levels and contamination degrees of impulsive noise contaminating the predominantly Gaussian-distributed observations, are illustrated in

Table 5. The $NMS$ criterion (50) for different algorithms and various outlier statistics in (40).

	pdf
	$\delta =0.1$	$\delta =0.1$	$\delta =0.1$	$\delta =0.1$
	$p_0\left(\cdot \right)=C\left(0,1\right)$	$p_0\left(\cdot \right)=R\left(-5,5\right)$	$p_0\left(\cdot \right)=L\left(0,1\right)$	$p_0\left(\cdot \right)=N\left(0,10\right)$
Algorithm
A1	1.195	0.105	0.177	0.104
A2	0.082	0.019	0.062	0.029
A3	0.081	0.051	0.064	0.029

,

Table 6. Average mean square error norm for different algorithms as a function of the outlier probability, with $\widehat{\theta }\left(0\right)=0$, $\Gamma \left(0\right)=0.1I$, $k=1.5$, ${\sigma }_n^2=1$, $p_0\left(\cdot \right)=N\left(\cdot |0,10\right)$.

	pdf in (40)
	$\delta =0$	$\delta =0.02$	$\delta =0.03$	$\delta =0.05$	$\delta =0.1$0
Algorithm
A1	0.020	0.053	0.071	0.086	0.104
A2	0.040	0.026	0.030	0.028	0.029
A3	0.038	0.026	0.030	0.025	0.029

and

Table 7. Average mean square error norm for different outlier intensities with $\widehat{\theta }\left(0\right)=0$, $\Gamma \left(0\right)=0.1I$, $k=1.5$, ${\sigma }_n^2=1$, $\delta =0.1$ for different algorithms.

	pdf
	$p_0\left(\cdot \right)=N\left(\cdot |0,3\right)$	$p_0\left(\cdot \right)=N\left(\cdot |0,6\right)$	$p_0\left(\cdot \right)=N\left(\cdot |0,10\right)$
Algorithm
A1	0.049	0.083	0.117
A2	0.024	0.026	0.040
A3	0.024	0.027	0.040

, depicting the average square error norm—the NMSE in (50)—on the basis of 100 Monte Carlo trials.

In particular,

Table 8. Average mean square error norm for different algorithms, as a function of saturation level $k$, with $\delta =0.1$, $p_0\left(\cdot \right)=N\left(\cdot |0,10\right)$,$\theta \left(0\right)=0$, $\Gamma \left(0\right)=0.1I$.

	$\mathit{k}$
	$k=1.0$	$k=1.5$	$k=2.0$	$k=3.0$	$k=4.0$
Algorithm
A2	0.030	0.028	0.028	0.033	0.042
A3	0.030	0.028	0.028	0.032	0.041

shows the dependence of the estimate quality on the tuning parameter $k\left(\delta \right)$ characterizing the nonlinear influence function ${\psi }_{*}$ in (34). Since the $\delta$-contamination degree in (40) is not known in practice, one has to adopt the $k$-parameter value in advance (see Table 1).

The simulation results confirm the previous conclusions drawn from the asymptotic theoretical analysis (see Table 1, Table 2 and Table 3). In this sense, the linear RLS algorithm A1 under purely Gaussian-distributed observations with $\delta =0$ in (40) ensures slightly better estimation quality than the AML-based recursive nonlinear identification algorithms, A2 and A3, optimal on the class in (40) (see Table 2 and Table 6). On other hand, algorithms A2 and A3 exhibit practical robustness, meaning that these algorithms are very effective in suppressing the effects of different types of outlying data points under the contamination degree $0<\delta \le 0.2$ (see Table 3, Table 5, Table 6 and Table 7). Moreover, algorithms A2 and A3 produce comparable results, presenting a good balance between estimation quality and insensitivity to multiple outliers (see Table 5, Table 6 and Table 7). Furthermore, Table 8 shows that the nonlinear algorithms A2 and A3 are rather insensitive to the value of the saturation level $k$ belonging to the interval $1<k\le 3$.

The good results obtained by algorithms A2 and A3 are due to the adequate nonlinear processing of prediction errors through the 1.5 Huber M-robust influence function ${\psi }_{*}$ in (43) within the parameter update recursion (14). This estimation procedure is also combined with the robust recursive scheme to calculate the gain matrix $\Gamma \left(i\right)$ in (16), based on either the fixed factor ${\alpha }_{*}$ in (46) or the variable factor ${\alpha }_k$ in (34). This, in turn, keeps the gain matrix $\Gamma \left(k\right)$ at values high enough to ensure the fast convergence of the estimates, but also at small enough values to ensure low sensitivity to outliers. In contrast, the RLS-type algorithm A1 is inferior to the nonlinear algorithms A2 and A3 in the presence of outliers corrupting the predominantly Gaussian-distributed data.

However, algorithms A2 and A3, which are optimal on the class in (40), are nonlinear; hence, their parameter estimates may be highly sensitive to the initial choices of $\widehat{\theta }\left(0\right)$ and $\Gamma \left(0\right)$. In general, Algorithm A3 converges slowly for higher values of $\Gamma \left(0\right)$ due to the influence of the dynamics of the variable factor ${\alpha }_k$. This factor represents the current slope of the ${\psi }_{*}$-nonlinearity in (43), which is further used to calculate the gain matrix $\Gamma \left(k\right)$ in (16). Thus, the large residual samples in the initial steps, induced by the initial value $\Gamma \left(0\right)$, lead to a very slow decrease in the gain $\Gamma \left(k\right)$ since the ${\psi }_{*}$-nonlinearity operates in the saturation mode, causing the current slope ${\alpha }_k$ to be close to zero. This, in turn, ensures a cumulative effect, resulting further in the slow convergence of the parameter estimates. In contrast, Algorithm A2 is found to be relatively insensitive to the value of $\Gamma \left(0\right)$ due to the effect of the fixed factor $\alpha$. This factor represents the average slope of ${\psi }_{*}$-nonlinearity in (43) and is further used to calculate the gain $\Gamma \left(k\right)$ in (16), resulting in the smooth dynamics of the gain $\Gamma$. This, in turn, reduces the influence of the initial values on the quality of parameter estimation.

In general, the problem related to the choice of the initial conditions can be circumvented by using good starting values $\widehat{\theta }\left(0\right)$ and $\Gamma \left(0\right)$, generated, for example, by a non-recursive Huber M-robust parameter estimator [15,23,24,25,38].

Finally, the computational complexity of the developed nonlinear algorithms A2 and A3 is slightly higher than that of the linear Algorithm A1. This result is achieved at the cost of limiting the condition related to the choice of the ARX model characterized by zero-mean wide-sense stationary process noise.

8. Conclusions

This paper has addressed the problem of robust recursive parameter identification for linear discrete-time stochastic systems operating under uncertain, potentially non-Gaussian disturbance statistics. To this end, a new class of nonlinear recursive algorithms of an approximate Newton–Raphson type for parameter identification in a linear discrete-time dynamic stochastic system, excited by zero-mean white stochastic disturbances, has been developed. Strong consistency and the convergence of the asymptotic estimation errors to a zero-mean normal pdf have been established by utilizing the martingale theory approach. Starting from the Cramér–Rao theorem and the derived covariance matrix of the asymptotic estimation error, the ML-type optimal nonlinear residual transformation under a fully known noise pdf is derived. This, in turn, leads to an asymptotically efficient ML-based recursive identification algorithm, with the asymptotic estimation error covariance matrix achieving the Cramér–Rao lower bound. The link between the ML-based optimal nonlinearity and the gain matrix calculation is established through the statistical linearization coefficient, representing the expected slope of the nonlinearity. The case of disturbance uncertainty has been considered by utilizing the available prior statistical information to specify a class of pdfs to which the real unknown observation noise pdf belongs, leading to the ML-based recursive parameter identification algorithm that is optimal on a class. The proposed design provides the minimum asymptotic estimation error covariance matrix under the least favorable or worst-case pdf within the adopted class; this covariance matrix is simultaneously the maximum one within the class. Unfortunately, the min–max optimal design may be indeterminable, since an exact solution for the worst-case pdf does not generally exist. In particular, it exists only for static regression plants and FIR linear dynamic stochastic systems. The matrix optimization procedure for finding the worst-case pdf reduces to minimizing the scalar Fisher information within the specified class. Furthermore, in the case of IIR linear dynamic systems, the exact numerical solution for the worst-case pdf exists only for a class of zero-mean symmetric pdfs with finite variance. Therefore, the derived worst-case pdf solution, minimizing the Fisher information, represents an approximate optimal worst-case pdf. This, in turn, leads to an approximate ML (AML) design on a class. In addition, the ML-based optimal- and AML-based optimal-on-a-class algorithms have the same recursive nonlinear structure as the Newton–Raphson-type method, where the only difference is in the optimization procedure to find either the optimal or suboptimal worst-case one. Moreover, the proposed AML-based design on a class is step-by-step optimal, minimizing the conditional estimation error covariance matrix at each stage under the suboptimal worst-case pdf. In particular, a class of $\delta$-contaminated Gaussian pdfs is of special importance in practice, since various scientific and industrial real measurements contain unavoidable outliers, corrupting the predominantly Gaussian-distributed measurements. In this sense, the AML-based suboptimal nonlinearity on the above class coincides with the influence functions of the Huber M-robust estimation of the location parameter, which also makes the AML-based design on the class robust in the practical sense, involving both efficiency and resistance robustness properties. However, the developed M-robustified version of the AML-based optimal on the given class design is still indeterminable, since the statistical linearization coefficient in Newton–Raphson-type recursion represents the unknown expected slope of the Huber M-robust influence function. A reasonable approach is to calculate the mathematical expectation by utilizing the AML-based suboptimal worst-case pdf, resulting in a fixed coefficient. Additionally, since such an approximation requires exact knowledge of the $\delta$-contamination degree, a more suitable approach is to approximate the expectation via the current measurement residual realization, producing a variable coefficient.

Simulation results have shown that the proposed M-robustified AML-based optimal-on-a-class design ensures a good balance between estimation quality and insensitivity to multiple outliers, up to a contamination degree no greater than twenty percent. For higher levels of outliers, the specified class of $\delta$-contaminated Gaussian pdfs is no longer adequate.

It is worth relating the proposed design to the ML-type parameter estimation procedures commonly used for ARMA, ARIMA, ARIMAX and GARCH models. In the ARMA(X) case, the moving-average noise term makes the one-step predictor nonlinear in the parameters, so that the conditional ML estimate is computed by iterative Gauss–Newton-type optimization or, recursively, by the recursive prediction error method (RPEM) [11,12]. The ARIMA(X) family additionally treats nonstationarity through differencing, while GARCH models describe conditionally heteroskedastic disturbances, estimated in practice by the quasi-ML approach. All these procedures rely on the Gaussian, or conditionally Gaussian, likelihood and are therefore sensitive to heavy-tailed disturbances and outliers in the same way as the RLS algorithm analyzed here. The ARX equation error structure with i.i.d. disturbances, adopted in this paper, represents the case in which the regression vector is measurable with respect to $\sigma$-algebra generated by the past input–output data, enabling the rigorous martingale-based convergence analysis, as well as the explicit M-robustified AML design on a class. Within this framework, RPEM essentially reduces to the RLS algorithm, so that the presented comparisons implicitly cover the corresponding ML-type benchmark.

Further investigations should be oriented towards designing an AML-based recursive identification algorithm, optimal on a class, under levels of prior information on stochastic disturbances that differ from those assumed here, involving correlated noise within the ARMA and ARMAX model structures (e.g., by M-robustifying the RPEM recursion); conditionally heteroskedastic disturbances of the GARCH type, with unequally distributed observations; asymmetric noise pdfs, etc. Some preliminary results may be found in the statistical literature, but their application to the dynamic system identification problem requires further work.