Kernel Identification of Non-Linear Systems with General Structure

Abstract

In the paper we deal with the problem of non-linear dynamic system identification in the presence of random noise. The class of considered systems is relatively general, in the sense that it is not limited to block-oriented structures such as Hammerstein or Wiener models. It is shown that the proposed algorithm can be generalized for two-stage strategy. In step 1 (non-parametric) the system is approximated by multi-dimensional regression functions for a given set of excitations, treated as representative set of points in multi-dimensional space. ‘Curse of dimensionality problem’ is solved by using specific (quantized or periodic) input sequences. Next, in step 2, non-parametric estimates can be plugged into least squares criterion and support model selection and estimation of system parameters. The proposed strategy allows decomposition of the identification problem, which can be of crucial meaning from the numerical point of view. The “estimation points” in step 1 are selected to ensure good task conditioning in step 2. Moreover, non-parametric procedure plays the role of data compression. We discuss the problem of selection of the scale of non-parametric model, and analyze asymptotic properties of the method. Also, the results of simple simulation are presented, to illustrate functioning of the method. Finally, the proposed method is successfully applied in Differential Scanning Calorimeter (DSC) to analyze aging processes in chalcogenide glasses.

1. Introduction

The problem of non-linear system modeling has been intensively examined over the past four decades. Owing to many potential applications (see [1]) and interdisciplinary scope of the topic, both scientists and engineers look for more precise and numerically efficient identification algorithms. First attempts at generalization of linear system identification theory for non-linear models were based on Volterra series representation ([2]). Traditional Volterra series-based approach leads to relatively high numerical complexity, which is often not acceptable from practical point of view. To cope with this problem regularization or tensor network techniques have been proposed recently ([3,4]). However, strong restrictions are imposed on the system characteristics (e.g., smoothness of nonlinearity, and short memory of the dynamics). Alternatively, the concept of block-oriented models was introduced ([5]). It was assumed that the system can be represented, or approximated with satisfactory accuracy, with the use of structural models including interconnected elementary blocks of two types—linear dynamics and static nonlinearities ([6]). The most popular structures in this class are Hammerstein system (see e.g., [7,8,9,10,11,12]) with static nonlinearity followed by linear dynamics, and the Wiener system [13,14,15,16,17,18]) including the same blocks connected reversely.

Traditionally, identification method needs two kinds of knowledge—set of input–output measurements and a priori parametric formula describing system characteristics and including finite number of unknown parameters ([6,19]). Usually, the polynomial model of static characteristics and the difference equation with known orders are assumed. This convention leads to relatively fast convergence of parameter estimates, but the risk of systematic approximation error appears, if the assumed model is not correct. As an alternative, the theory of non-parametric system identification ([20,21,22,23]) was proposed to solve this problem. The algorithms work under mild prior restrictions, such as stability of linear dynamic block and local continuity of static non-linear characteristics. Although the estimates converge to the true characteristics, the rate of convergence is relatively slow in practice, as a consequence of assumptions relaxation.

This paper represents the idea of combined, i.e., parametric-non-parametric, approach (see e.g., [24,25,26,27,28,29,30]), in which both parametric and non-parametric algorithms support each other to achieve the best possible results of identification for moderate number of measurements and to guarantee the asymptotic consistency (i.e., convergence to the true system characteristics), when the number of observations tends to infinity. Since the preliminary step of structure selection is treated rather cursorily in the literature, generalizations of the approach towards wider classes of systems seem to be of high importance from practical point of view. The paper was also motivated by the project of Differential Scanning Calorimeter ([31]) developed in the team, and particularly, modeling of heating process for examination the properties of chalcogenide glasses.

Main contribution of the paper lays in the following aspects:

• proposed identification algorithm is run without any prior knowledge about the system structure and parametric representation of nonlinearity,
• non-parametric multi-dimensional kernel regression estimate was generalized for modeling of non-linear dynamic systems, and the dimensionality problem was solved by using special input sequences,
• the scheme elaborated in the paper was successfully applied in Differential Scanning Calorimeter for testing parameters of chalcogenide glasses.

The paper is organized as follows. In Section 2 the general class of considered systems and the identification problem is formulated in detail. Next, in Section 3.1, purely non-parametric, regression-based approach is presented, and its disadvantages are discussed. Then, to cope with dimensionality problem, the idea of some specific input sequences is presented in Section 3.2. Owing to that, the system characteristics are identified only for some selected points, but the convergence is much faster. The idea of combined, two-stage strategy is introduced in Section 4. It allows use of prior knowledge to expand the model on the whole input space. Also, the results of simple simulation are included in Section 5 to illustrate and discuss some practical aspects of the approach. Finally, in Section 6, the algorithm is successfully applied in Differential Scanning Calorimeter to model aging properties of modern materials (chalcogenide glasses).

2. Problem Statement

2.1. Class of Systems

We consider discrete-time non-linear dynamic system with general representation

$$y_k=\mathcal{F}\left({\left\{u_{k-i}\right\}}_{i=0}^{\infty }\right)+z_k,$$

(1)

where ${\left\{u_{k-i}\right\}}_{i=0}^{\infty }$ is bounded random input sequence ($\left|u_k\right|<u_{max}$), $z_k$ is zero mean random disturbance. The transformation $\mathcal{F}\left(\right)$ is Lipschitz with respect to all arguments, and has the property of exponential forgetting (fading memory), i.e., if we put $u_{k-i}=0$ for $i\ge s$ and define the cut-off sequence as

$${\overline{u}}_{k-i}\triangleq \left\{\begin{array}{c}{u}_{k-i},\mathrm{as}i=0,1,...,s-1\\ 0,\mathrm{as}i\ge s\end{array}\right.,$$

(2)

We assume that it holds that

$$\Delta \left(s\right)\triangleq \left|\mathcal{F}\left({\left\{u_{k-i}\right\}}_{i=0}^{\infty }\right)-\mathcal{F}\left({\left\{{\overline{u}}_{k-i}\right\}}_{i=0}^{\infty }\right)\right|\le c{\lambda }^s,$$

(3)

with some unknown $c=\mathrm{const}$, and $0<\lambda <1$. Similar class of fading memory systems, in which the output depends less and less on historical inputs, was considered in [32]. The goal is to identify the system (build the model $\widehat{\mathcal{F}}\left(\right)$) using the sequence of N input–output measurements ${\left\{\left(u_k,y_k\right)\right\}}_{k=1}^N$.

In considered system class, hysteresis is not admitted.

2.2. Examples

In this section, we show that some systems (popular in applications) fall into above description as special cases.

2.2.1. Hammerstein System

For Hammerstein system (see Figure 1), described by the equation

$$y_k=\sum _{i=0}^{\infty }{\gamma }_i\mu \left(u_{k-i}\right)+z_k$$

(4)

with the Lipschitz non-linear characteristic, i.e., such that

$$\left|\mu \left(u_a\right)-\mu \left(u_b\right)\right|\le l\left|u_a-u_b\right|,$$

(5)

and asymptotically stable dynamics, i.e., $\left|{\gamma }_i\right|\le c_H{\lambda }^i$, we get

$$\Delta \left(s\right)=\left|\sum _{i=s}^{\infty }{\gamma }_i\left(\mu \left(u_{k-i}\right)-\mu \left(0\right)\right)\right|\le l{u}_{max}\left|\sum _{i=s}^{\infty }{\gamma }_i\right|\le c{\lambda }^s.$$

(6)

Figure 1. Hammerstein system.

2.2.2. Wiener System

Analogously, for Wiener system (Figure 2), where the stable linear dynamics is followed by the Lipschitz static non-linear block

$$y_k=\mu \left(\sum _{i=0}^{\infty }{\lambda }_i{u}_{k-i}\right)+z_{k,}$$

(7)

Figure 2. Wiener System.

We get

$$\begin{array}{ccc}\hfill \Delta \left(s\right)& =& \left|\mu \left(\sum _{i=0}^{\infty }{\gamma }_i{u}_{k-i}\right)-\mu \left(\sum _{i=0}^{s-1}{\gamma }_i{u}_{k-i}\right)\right|\hfill \end{array}$$

(8)

$$\begin{array}{ccc}& \le & l\left|\sum _{i=s}^{\infty }{\gamma }_i{u}_{k-i}\right|<c{\lambda }^s.\hfill \end{array}$$

(9)

Remark 1.

Analogously, it can be simply shown that also Wiener–Hammerstein (L–N–L) and Hammerstein–Wiener (N–L–N) sandwich systems belong to the assumed class.

2.2.3. Finite Memory Bilinear System

Another important and often met in application special case is the bilinear system with finite order m. It is described by the formula

$$y_k=\sum _{l_0,l_1,...,l_{m-1}:l_{i\ge 0}\cap {\sum }_i{l}_i\le m}{c}_{l_0,l_1,...,l_{m-1}}\prod _{i=0}^{m-i}{u}_{k-i}^{l_i}+z_k$$

(10)

i.e., for $m=1$ we get

$$y_k=c_{0,0}+c_{1,0}{u}_k+c_{0,1}{u}_{k-1},$$

(11)

for $m=2$ we have that

$$y_k=c_{0,0}+c_{1,0}{u}_k+c_{0,1}{u}_{k-1}+c_{1,1}{u}_k{u}_{k-1}+c_{2,0}{u}_k^2+c_{0,2}{u}_{k-1}^2.$$

(12)

Since $y_k$ does not depend on $u_{k-m},u_{k-m-1},...$, for $s\ge m$ we get $\mathcal{F}\left({\left\{u_{k-i}\right\}}_{i=0}^{\infty }\right)=\mathcal{F}\left({\left\{{\overline{u}}_{k-i}\right\}}_{i=0}^{\infty }\right)$, and obviously $\Delta \left(s\right)=0$.

Considered example falls into the more general class of Volterra representation. Presented approach works without the knowledge of parametric representation. As regards applicability to the class (1), for $m<\infty$ we have finite memory system, i.e., $\Delta \left(s\right)=0$, as $s>m$. Moreover, since the input is assumed to be bounded ($\left|u_k\right|<u_{max}$), resulting mapping $\mathcal{F}\left(\right)$ fulfills Lipschitz condition (as ordinary polynomial on compact support).

3. Non-Parametric Regression

3.1. General Overview

Let us introduce the s-dimensional input regressor

$${\mathbf{u}}_k^{\left(s\right)}={\left(u_k,u_{k-1},...,u_{k-(s-1)}\right)}^T,$$

(13)

and the regression function

$$R_s\left({\mathbf{x}}^{\left(s\right)}\right)=E\left(y_k|{\mathbf{u}}_k^{\left(s\right)}={\mathbf{x}}^{\left(s\right)}\right),$$

(14)

with the argument vector

$${\mathbf{x}}^{\left(s\right)}={\left(x_0,x_1,...,x_{s-1}\right)}^T.$$

(15)

In particular, for $s=1$ we get

$$R_1\left(x_0\right)=E\left(y_k|u_k=x_0\right),$$

(16)

and for $s=2$

$$R_2(x_0,x_1)=E\left(y_k|u_k=x_0\cap u_{k-1}=x_1\right).$$

(17)

The non-parametric kernel estimate ([20,21,22,23,33]) of $R_s\left({\mathbf{x}}^{\left(s\right)}\right)$ has the following form

$${\widehat{R}}_s\left({\mathbf{x}}^{\left(s\right)}\right)=\frac{{\sum }_{k=1}^N{y}_{k}K\left(\frac{∥{\mathbf{u}}_k^{\left(s\right)}-{\mathbf{x}}^{\left(s\right)}∥}{h}\right)}{{\sum }_{k=1}^{N}K\left(\frac{∥{\mathbf{u}}_k^{\left(s\right)}-{\mathbf{x}}^{\left(s\right)}∥}{h}\right)},$$

(18)

where $∥·∥$ denotes Euclidean norm, $K\left(\right)$ plays the role of kernel function, e.g.,

$$K\left(v\right)=\left\{\begin{array}{c}1,\mathrm{as}\left|v\right|\le 1\\ 0,\mathrm{as}\left|v\right|>1\end{array}\right.,$$

(19)

and h is a bandwidth parameter, responsible for the balance between bias and variance of the estimate. The class of possible kernels can obviously be generalized. Nevertheless, previous experiences shows that the kind of kernel function used for estimation is of secondary importance, whereas behavior of $h\left(N\right)$ with respect to N is fundamental. We limited the presentation to Parzen (window) kernel for clarity of exposition. It fulfills all general assumptions made for kernels, i.e., it is even, non-negative and square integrable. The system (1) is thus approximated by the model

$${\widehat{R}}_s\left(u_k,u_{k-1},...,u_{k-(s-1)}\right),$$

(20)

and s can be interpreted as its complexity. Obviously, both $h=h\left(N\right)$ and $s=s\left(N\right)$ need to be set depending on the number of measurements N. Observing that

$$\mathcal{F}\left({\left\{u_{k-i}\right\}}_{i=0}^{\infty }\right)=R_{\infty }\left({\mathbf{u}}_k^{(\infty )}\right),$$

(21)

the mean squared error of the model ${\widehat{R}}_s\left({\mathbf{u}}_k^{\left(s\right)}\right)$ can be expressed as follows

$$\begin{array}{c}MSE\left({\widehat{R}}_s\left({\mathbf{u}}_k^{\left(s\right)}\right)\right)=E{\left({\widehat{R}}_s\left({\mathbf{u}}_k^{\left(s\right)}\right)-\mathcal{F}\left({\left\{u_{k-i}\right\}}_{i=0}^{\infty }\right)\right)}^2=E{\left({\widehat{R}}_s\left({\mathbf{u}}_k^{\left(s\right)}\right)-R_{\infty }\left({\mathbf{u}}_k^{(\infty )}\right)\right)}^2,\end{array}$$

and introducing the true finite-dimensional regression function $R_s\left({\mathbf{u}}_k^{\left(s\right)}\right)$ we get

$$\begin{array}{c}MSE\left({\widehat{R}}_s\left({\mathbf{u}}_k^{\left(s\right)}\right)\right)=E\left\{\left({\widehat{R}}_s\left({\mathbf{u}}_k^{\left(s\right)}\right)-R_s\left({\mathbf{u}}_k^{\left(s\right)}\right)\right)\right.+{\left.\left(R_s\left({\mathbf{u}}_k^{\left(s\right)}\right)-R_{\infty }\left({\mathbf{u}}_k^{(\infty )}\right)\right)\right\}}^2=\\ E{\left\{{\widehat{R}}_s\left({\mathbf{u}}_k^{\left(s\right)}\right)-R_s\left({\mathbf{u}}_k^{\left(s\right)}\right)\right\}}^2+E{\left\{R_s\left({\mathbf{u}}_k^{\left(s\right)}\right)-R_{\infty }\left({\mathbf{u}}_k^{(\infty )}\right)\right\}}^2+\\ +2E\left\{\left({\widehat{R}}_s\left({\mathbf{u}}_k^{\left(s\right)}\right)-R_s\left({\mathbf{u}}_k^{\left(s\right)}\right)\right)\left(R_s\left({\mathbf{u}}_k^{\left(s\right)}\right)-R_{\infty }\left({\mathbf{u}}_k^{(\infty )}\right)\right)\right\}.\end{array}$$

Since both $E{\left\{R_s\left({\mathbf{u}}_k^{\left(s\right)}\right)-R_{\infty }\left({\mathbf{u}}_k^{(\infty )}\right)\right\}}^2\to 0$ as $N\to \infty$ and $\left(R_s\left({\mathbf{u}}_k^{\left(s\right)}\right)-R_{\infty }\left({\mathbf{u}}_k^{(\infty )}\right)\right)\to 0$ as $N\to \infty$, these components can be set arbitrarily small by using appropriate scale s. Owing to above, for fixed s we focus on the first component of the MSE error of the form

$$\begin{array}{ccc}\hfill ERR& =& E{\left\{{\widehat{R}}_s\left({\mathbf{u}}_k^{\left(s\right)}\right)-R_s\left({\mathbf{u}}_k^{\left(s\right)}\right)\right\}}^2={bias}^2{\widehat{R}}_s\left({\mathbf{u}}_k^{\left(s\right)}\right)+var{\widehat{R}}_s\left({\mathbf{u}}_k^{\left(s\right)}\right),\hfill \end{array}$$

(22)

where

$$\begin{array}{ccc}\hfill bias{\widehat{R}}_s\left({\mathbf{u}}_k^{\left(s\right)}\right)& \triangleq & E{\widehat{R}}_s\left({\mathbf{u}}_k^{\left(s\right)}\right)-R_s\left({\mathbf{u}}_k^{\left(s\right)}\right),\hfill \end{array}$$

(23)

$$\begin{array}{ccc}\hfill var{\widehat{R}}_s\left({\mathbf{u}}_k^{\left(s\right)}\right)& \triangleq & E{\left\{{\widehat{R}}_s\left({\mathbf{u}}_k^{\left(s\right)}\right)-E{\widehat{R}}_s\left({\mathbf{u}}_k^{\left(s\right)}\right)\right\}}^2.\hfill \end{array}$$

(24)

It can simply be shown that

$$\begin{array}{ccc}\hfill bias{\widehat{R}}_s\left({\mathbf{x}}^{\left(s\right)}\right)& =& o\left(h\left(N\right)\right),\hfill \end{array}$$

(25)

$$\begin{array}{ccc}\hfill {bias}^2{\widehat{R}}_s\left({\mathbf{x}}^{\left(s\right)}\right)& =& o\left(h^2\left(N\right)\right),\hfill \end{array}$$

(26)

The bias order follows directly from Lipschitz condition, and the fact that $∥{\mathbf{u}}_k^{\left(s\right)}-{\mathbf{x}}_{\left[p\right]}^{\left(s\right)}∥\le h$ for all k’s selected by kernel. Moreover,

$$\begin{array}{ccc}\hfill var{\widehat{R}}_s\left({\mathbf{x}}^{\left(s\right)}\right)& =& o\left(\frac{1}{N{h}^s\left(N\right)}\right).\hfill \end{array}$$

(27)

For window kernel, Lipschitz function $\mathcal{F}$, and strictly positive input probability density function around the estimation point, probability of selection in s-dimensional space can be obviously evaluated from below by $c{h}^s$, where c is some constant. Hence, expected number of successes is proportional to $Nc{h}^s$ (not less than). The variance order is thus a simple consequence of Wald’s identity. Hence, to assure the convergence ${\widehat{R}}_s\left({\mathbf{x}}^{\left(s\right)}\right)\to R_s\left({\mathbf{x}}^{\left(s\right)}\right)$, as $N\to \infty$ in the mean square sense, the following conditions must be fulfilled

$$h\to 0\mathrm{and}N{h}^s\to \infty ,\mathrm{as}N\to \infty ,$$

(28)

which leads to typical setting

$$h\left(N\right)\sim N^{-\alpha },\mathrm{where}\alpha \in \left(0,\frac{1}{s}\right).$$

(29)

To obtain the best asymptotic trade-off between squared bias and variance and comparing its orders we get

$$\begin{array}{ccc}\hfill h^2\left(N\right)& =& \frac{1}{N{h}^s\left(N\right)},\hfill \end{array}$$

(30)

$$\begin{array}{ccc}\hfill h^{s+2}\left(N\right)& =& \frac{1}{N},\hfill \end{array}$$

(31)

$$\begin{array}{ccc}\hfill h_{opt}\left(N\right)& \sim & N^{-\frac{1}{s+2}},\hfill \end{array}$$

(32)

$$\begin{array}{ccc}\hfill ERR& \sim & N^{-\frac{2}{s+2}}.\hfill \end{array}$$

(33)

Moreover, to assure the balance between the estimation error and approximation error of order $o\left({\lambda }^s\right)$, connected with neglecting the tail ${\left\{u_{k-i}\right\}}_{i=s}^{\infty }$ we get

$$\begin{array}{ccc}\hfill N^{-\frac{2}{s\left(N\right)+2}}& =& {\lambda }^{s\left(N\right)},\hfill \end{array}$$

(34)

$$\begin{array}{ccc}\hfill -\frac{2}{s\left(N\right)+2}logN& =& s\left(N\right)log\lambda ,\hfill \end{array}$$

(35)

$$\begin{array}{ccc}\hfill \frac{-2}{s\left(N\right)\left(s\left(N\right)+2\right)}& =& \frac{log\lambda }{logN},\hfill \end{array}$$

(36)

$$\begin{array}{ccc}\hfill s^2\left(N\right)+2s\left(N\right)& =& \frac{2}{log\frac{1}{\lambda }}logN,\hfill \end{array}$$

(37)

where $\frac{2}{log\frac{1}{\lambda }}=\mathit{const}$. Owing to above, the scale $s\left(N\right)$ must not be faster than $\sqrt{logN}$, i.e.,

$$s_{opt}\left(N\right)=\mathcal{O}\left(\sqrt{logN}\right).$$

(38)

which illustrates slowness of admissible model complexity increasing in general case. The property (38) commonly known as “curse of dimensionality problem” illustrates the main drawback of multi-dimensional non-parametric regression approach to system modeling in traditional form. The reason is that probability of kernel selection

$$P\left\{K\left(\frac{∥{\mathbf{u}}_k^{\left(s\right)}-{\mathbf{x}}^{\left(s\right)}∥}{h}\right)=1\right\}=P\left\{∥{\mathbf{u}}_k^{\left(s\right)}-{\mathbf{x}}^{\left(s\right)}∥\le h\right\}\sim h^s$$

(39)

decreases rapidly when s grows large. We also refer the reader to the proof of Theorem 3 in [26], where a detailed discussion concerning an analogous problem can be found.

3.2. Dimension Reduction

To cope with the problem shown in (39) we consider two cases of some specific input excitation processes to speed up the rate of convergence.

3.2.1. Discrete Input

In case 1 we assume that in each s-element input sub-sequence $u_k,u_{k+1},...,u_{k+s-1}$, there exists d inputs with discrete distribution on finite set of possible realizations. Consequently, all the points ${\mathbf{u}}_k^{\left(s\right)}\in R^s$ lay on a finite number of separable and compact subspaces with the internal dimension

$$s^{*}=s-d,$$

(40)

and for ${\mathbf{x}}^{\left(s\right)}={\mathbf{u}}_k^{\left(s\right)}$ we have

$$ERR\sim N^{-\frac{2}{s^{*}+2}}.$$

(41)

For each measurement point probability of kernel selection behaves like $c{h}^{s^{*}}$, where $s^{*}$, $(s^{*}<s)$, is internal dimension of this subspace. In particular, for $d=s$ (all input variables quantized) we get $ERR\sim N^{-1}$. The sets of possible input sequences for $s=2$ are illustrated in Figure 3 and Figure 4.

Figure 3. Input space for $s=2$ and $d=1$, (a) k odd, (b) k even.

Figure 4. Input space for $s=2$ and $d=2$.

3.2.2. Periodic Input

In case 2 we assume the input is periodic with the period $N_0$, i.e., $u_k=u_{k+N_0}$ for each $k=1,2,...,N-N_0$. Then, the value of the regressor ${\mathbf{u}}_k^{\left(s\right)}$ (see (13)) evaluates to one of $N_0$ distinct points in $R^s$, $n\in Z$,

$${\mathbf{x}}_{\left[1\right]}^{\left(s\right)}=\left[\begin{array}{c}{u}_s\\ u_{s-1}\\ ⋮\\ u_1\end{array}\right],...,{\mathbf{x}}_{\left[N_0\right]}^{\left(s\right)}=\left[\begin{array}{c}{u}_{N_0+s-1}\\ u_{N_0+s-2}\\ ⋮\\ u_{N_0}\end{array}\right]$$

(42)

with probabilities

$$P\left\{{\mathbf{u}}_k^{\left(s\right)}={\mathbf{x}}_{\left[p\right]}^{\left(s\right)}\right\}=\frac{1}{N_0},p=1,2,...,N_0$$

(43)

Measurements are uniformly distributed on the finite set of $N_0$ distinct points

$${\mathbf{x}}_{\left[1\right]}^{\left(s\right)},...,{\mathbf{x}}_{\left[p\right]}^{\left(s\right)},...{\mathbf{x}}_{\left[N_0\right]}^{\left(s\right)}$$

(44)

Narrowing of $h\left(N\right)$ does not affect the kernel estimator asymptotically (i.e., $s^{*}=0$). Consequently, we get the best possible convergence rate $ERR\sim N^{-1}$. However, it must be admitted that estimators are calculated only for finite number of points, and, increasing $N_0$ causes increase of variance of the regression estimator for particular points ${\mathbf{x}}_{\left[p\right]}^{\left(s\right)}$ (as the number of selected data is of order $N/N_0$).

4. Hybrid/Combined Parametric-Non-Parametric Approach

Since the special input excitations allows for fast recovering the system characteristics only in some points, additional prior knowledge is needed to extend the model for arbitrary process $\left\{u_k\right\}$. In this section we assume that the transformation $\mathcal{F}\left({\left\{u_{k-i}\right\}}_{i=0}^{\infty }\right)$ belongs to the given (a priori known), finite dimensional class of systems $F\left({\mathbf{u}}_k^{\left(s\right)},\theta \right)$

$$\mathcal{F}\left({\left\{u_{k-i}\right\}}_{i=0}^{\infty }\right)\subset F\left({\mathbf{u}}_k^{\left(s\right)},\theta \right)$$

(45)

with unknown parameter vector $\theta$. In the proposed methodology, one of the input excitations described in Section 4 is applied. The system is identified on the finite set of $N_0$ representative points ${\mathbf{x}}_{\left[1\right]}^{\left(s\right)},{\mathbf{x}}_{\left[2\right]}^{\left(s\right)},...,{\mathbf{x}}_{\left[p\right]}^{\left(s\right)},...,{\mathbf{x}}_{\left[N_0\right]}^{\left(s\right)}$, where ${\mathbf{x}}_{\left[p\right]}^{\left(s\right)}\in R^s$, and $p=1,2,...,N_0$. Let us denote by ${\theta }^{*}$—the true and unknown vector of system parameters. We assume that ${\theta }^{*}$ is identifiable, i.e., the following property holds

$$\mathcal{F}\left({\left\{u_{k-i}\right\}}_{i=0}^{\infty }\right)=F\left({\mathbf{u}}_k^{\left(s\right)},\theta \right)⟺\theta ={\theta }^{*}.$$

(46)

Moreover, let the quality index

$$Q\left(\theta \right)=E{\left[y_k-F\left({\mathbf{u}}_k^{\left(s\right)},\theta \right)\right]}^2$$

(47)

be convex for $\theta \in \mathsf{\Xi }$, where $\mathsf{\Xi }$ is some neighborhood of the true ${\theta }^{*}$

$${\theta }^{*}=arg\underset{\theta }{min}Q\left(\theta \right).$$

(48)

The following two-step algorithm is proposed.

Step 1. (non-parametric) Using the input–output observations ${\left\{\left(u_k,y_k\right)\right\}}_{k=1}^N$ for $p=1,2,...,N_0$ compute the estimates

$${\widehat{R}}_s\left({\mathbf{x}}_{\left[1\right]}^{\left(s\right)}\right),...,{\widehat{R}}_s\left({\mathbf{x}}_{\left[p\right]}^{\left(s\right)}\right),...,{\widehat{R}}_s\left({\mathbf{x}}_{\left[N_0\right]}^{\left(s\right)}\right),$$

(49)

Step 2. (parametric) Minimize the empirical version of the least squares criterion (47)

$$\widehat{\theta }=arg\underset{\theta }{min}\frac{1}{N_0}\sum _{p=1}^{N_0}{\left({\widehat{R}}_s\left({\mathbf{x}}_{\left[p\right]}^{\left(s\right)}\right)-F\left({\mathbf{x}}_{\left[p\right]}^{\left(s\right)},\theta \right)\right)}^2.$$

(50)

Lemma 1.

Let $F\left({\mathbf{u}}_k^{\left(s\right)},\theta \right)$ be Lipschitz with respect to all $u_{k-l}$’s included in ${\mathbf{u}}_k^{\left(s\right)}$ and all parameters in θ. If the error of non-parametric estimate behaves like $∥{\widehat{R}}_s\left({\mathbf{x}}_{\left[p\right]}^{\left(s\right)}\right)-F\left({\mathbf{x}}_{\left[p\right]}^{\left(s\right)},{\theta }^{*}\right)∥=O\left(N^{-\tau }\right)$ in the mean square sense, for all $p=1,2,...,N_0$, then

$$∥\widehat{\theta }-{\theta }^{*}∥=O\left(N^{-\tau }\right)$$

(51)

in the parametric step 2.

Proof. 

The property (51) can be proven following the lines of the proof of Theorem 1 in [25]. □

Remark 2.

Fulfillment of (46) and the method of non-linear optimization in (50) are strictly dependent on the specifics of the problem. In active experiment, when the input can be generated arbitrarily, appropriate selection of the points ${\left\{{\mathbf{x}}_{\left[p\right]}^{\left(s\right)}\right\}}_{p=1}^{N_0}$ can significantly simplify operations in Step 2 (see example below).

Example 1.

For the system

$$y_k=e^{{\theta }_1{u}_k}+{\theta }_2{u}_k{u}_{k-1}+z_k,$$

(52)

in step 1 we can estimate two-dimensional ($s=2$) regression function ${\widehat{R}}_2\left({\mathbf{x}}^{\left(2\right)}\right)$ in $N_0=2$ representative points ${\mathbf{x}}_{\left[1\right]}^{\left(2\right)}={\left(1,0\right)}^T$, and ${\mathbf{x}}_{\left[2\right]}^{\left(2\right)}={\left(1,1\right)}^T$, i.e., compute the pattern

$${\widehat{R}}_2\left({\mathbf{x}}_{\left[1\right]}^{\left(2\right)}\right),{\widehat{R}}_2\left({\mathbf{x}}_{\left[2\right]}^{\left(2\right)}\right).$$

(53)

Since the true values of the regression function are respectively $R_2\left({\mathbf{x}}_{\left[1\right]}^{\left(2\right)}\right)=e^{{\theta }_1}$ and $R_2\left({\mathbf{x}}_{\left[2\right]}^{\left(2\right)}\right)=e^{{\theta }_1}+{\theta }_2$, in step 2 we get trivial estimates of parameters

$$\begin{array}{ccc}\hfill {\widehat{\theta }}_1& =& log{\widehat{R}}_2\left({\mathbf{x}}_{\left[1\right]}^{\left(2\right)}\right),\hfill \end{array}$$

(54)

$$\begin{array}{ccc}\hfill {\widehat{\theta }}_2& =& {\widehat{R}}_2\left({\mathbf{x}}_{\left[2\right]}^{\left(2\right)}\right)-{\widehat{R}}_2\left({\mathbf{x}}_{\left[1\right]}^{\left(2\right)}\right).\hfill \end{array}$$

(55)

5. Simulation Example

To illustrate the proposed method, we simulated simple Wiener system (see Figure 2) with

$$\begin{array}{ccc}\hfill x_k& =& 0.5x_{k-1}+u_k\hfill \end{array}$$

(56)

$$\begin{array}{ccc}\hfill v_k& =& \mathrm{arctg}\left(x_k\right)\hfill \end{array}$$

(57)

$$\begin{array}{ccc}\hfill y_k& =& v_k+z_k\hfill \end{array}$$

(58)

excited by random process uniformly distributed on equidistant set of points

$$u_k\sim \left\{-1,-0.75,-0.5,-0.25,0,0.25,0.5,0.75,1\right\},$$

(59)

and uniformly distributed output disturbance

$$z_k\sim U\left[-0.1,0.1\right].$$

(60)

For $N={10}^4$ simulated input–output pairs ${\left\{\left(u_k,y_k\right)\right\}}_{k=1}^N$, the non-parametric models ${\widehat{R}}_s\left({\mathbf{u}}_k\right)$ were computed for $s=1,2,3$ and compared with respect to the following empirical error

$${\delta }_s=\frac{1}{N-s+1}\sum _{k=s}^N{\left(y_k-{\widehat{R}}_s\left({\mathbf{u}}_k^{\left(s\right)}\right)\right)}^2.$$

(61)

The results are presented in Table 1 and Figure 5 and Figure 6. Explicit derivation of the true finite order (2D or 3D) regression function is problematic, owing to the fact that the neglected part of input signal, i.e., the ‘tail’ connected with terms ${\left\{u_{k-\tau }\right\}}_{\tau =s+1}^{\infty }$, is transferred through the non-linear characteristic $arctg\left(\right)$. Figure 5 and Figure 6 illustrate non-parametric character and non-linear properties of the model, and give a general view on the shape of input–output relationship, which can be helpful for eventual parametrization. Quantized input (59) can be a good choice, when $s=\mathrm{const}<\infty$, and the non-parametric estimate ${\widehat{R}}_s\left({\mathbf{u}}_k^{\left(s\right)}\right)$ plays supporting role for non-linear least squares-based parameter estimation in step 2. Nevertheless, in purely non-parametric approach, i.e., for $s\left(N\right)\to \infty$, the number of possible realizations of ${\mathbf{u}}_k^{\left(s\right)}$ increases exponentially. In the considered example, for 9 points in (59), probability of kernel selection for each ${\mathbf{x}}^{\left(s\right)}={\mathbf{u}}_k^{\left(s\right)}$ behaves like $\sim 9^{-s}$ and the estimate becomes sensitive on the noise $z_k$.

Table 1. Mean squared errors ${\delta }_s$ of model outputs for $s=1,2,3$, compared to best linear approximation (BLA).

	${\mathit{\delta }}_1$	${\mathit{\delta }}_2$	${\mathit{\delta }}_3$
$\widehat{R_s}$	0.0656	0.0279	0.0137
BLA	0.0660	0.0304	0.0176

Figure 5. Two-dimensional regression-based model ${\widehat{R}}_2\left(x_0,x_1\right)$.

Figure 6. Three-dimensional regression-based model ${\widehat{R}}_3\left(x_0,x_1,x_2\right)$.

Table 1 illustrates reduction of error with respect to scale of the regression. The results are also compared with the best linear approximations $FIR\left(s\right)$ of Wiener system. We emphasize that improvement is achieved under mild prior restrictions, i.e., the non-linear model is built based on measurements knowledge only.

6. Application in Testing of Chalcogenide Glasses with the Use of DSC Method

In this section, we apply the proposed algorithm for identification of heating process in Differential Scanning Calorimeter [31].

6.1. Chalcogenide Glasses

Materials with non-linear optical properties play a key role in frequency conversion and optical switching. One of the most promising materials in this area are chalcogenide glasses, because of good non-linear, passive and active properties. They are considered to be optical medium for the fibers of the 21st century. Chalcogenide glass fibers transmit into the IR, hence they can have numerous potential applications in the civil, medical and military areas. The IR light sources, lasers and amplifiers developed using these phenomena will be very useful in many areas of industry. High-speed optical communication requires ultra-fast all-optical processing and switching capabilities. In DSC experiment energy (heat flow) in function of time or temperature could be established. The energy from an external source is required to set to zero the difference in temperature of the tested and reference samples. Both samples are heated or cooled in a controlled mode and both techniques enable the detection of thermal events observed in the physical or chemical transformation under the influence of the changing temperature in a specific manner. Owing to that, many thermodynamically important parameters can be established, e.g., a glass transition or softening temperature, melting temperature, and melting enthalpy. The results also allow observation of physical aging processes. The goal is to control temperature of heating module precisely and ensure linearity of it. It is planned to design Model Following Control (MFC) structure of system to optimize quality indexes of temperature controlling. Below we present the results of identification experiment.

6.2. Results of Experiment

Treating the sample temperature as system output $y_k$, and the power of the heating element as input $u_k$, the non-parametric multi-dimension regression model ${\widehat{R}}_s\left({\mathbf{u}}_k^{\left(s\right)}\right)$ was computed for $s=1,2,3,4$. The results are shown in Figure 7, Figure 8, Figure 9 and Figure 10.

Figure 7. Experimental data—1-dimensional regression.

Figure 8. Experimental data—2-dimensional regression.

Figure 9. Experimental data—3-dimensional regression.

Figure 10. Experimental data—4-dimensional regression.

Differential Scanning Calorimeter for chalcogenide glasses (built by members of the team), was first approximated by the linear model, and the results were not satisfying. To improve accuracy, Hammerstein model was applied, and the decision of model structure was made arbitrarily. To avoid the risk of bad parameterization the general approach presented in the paper was applied. The results are comparable, although obtained without making any restrictive assumptions about the block-oriented structure of the model.

In Table 2 resulting mean squared errors for various scales $s=1,2,3,4$ are shown. The strong point of the method is that asymptotically, as $s\left(N\right)\to \infty$, the model becomes free of approximation error, on the contrary to linear or block-oriented representation.

Table 2. Mean squared errors ${\delta }_s$ of model outputs for $s=1,2,3,4$.

	${\mathit{\delta }}_1$	${\mathit{\delta }}_2$	${\mathit{\delta }}_3$	${\mathit{\delta }}_4$
$\widehat{R_s}$ (presented method)	1017	477	232	169
BLA (Linear FIR(s))	1710	1331	1165	1114
Hammerstein polynomial (3rd order + FIR(s))	1102	553	296	202

The results have been compared to FIR(s) linear model and parametric Hammerstein model. Regarding non-parametric modeling of Hammerstein systems, proposed by Greblicki and Pawlak in the 1980s [20], their algorithms suffer from correlation of input, and they are not applicable here. On the other hand, for parametric Hammerstein model, the results are strongly dependent on the arbitrarily selected basis functions of nonlinearity. In our experiment we applied 3rd order polynomial function $\mu \left(\right)$ connected in cascade with FIR(s) linear dynamic filter. Table 2 shows that our method is more accurate, emphasizing that it works under mild prior restrictions.

7. Conclusions

The main contribution of the work lays in the fact that the model is built with lack of prior knowledge about the structure of the system and its characteristics. No decision of using particular Hammerstein or Wiener model is needed at the beginning to start the procedure. Obtained non-parametric estimators ${\widehat{R}}_s\left({\mathbf{x}}_{\left[p\right]}^{\left(s\right)}\right)$ can be eventually plugged into the least squares optimization criterion in step 2, to provide parametric representation of the relationship. Both parametric and non-parametric methods can be combined to design strategy, which includes advantages of both approaches. Step 1 (non-parametric) is run to estimate selected points of system characteristic. It is done effectively thanks to generation of specific input excitation (discrete or periodic), which allows to avoid the problem of high dimensionality. Moreover, appropriate selection of estimation points can significantly decrease level of difficulty of the non-linear optimization task in step 2. The rate of convergence of parameter estimate is the same as for non-parametric ones.

The scheme presented in the paper is universal for a broad class of systems including Hammerstein and Wiener structures, and their interconnections. Non-parametric data pre-filtering plays also the role of compression algorithm and the result of step 1 can be treated as the simplified pattern of system for eventual structure detection and selection of its best parametric model. Regression-based non-parametric model can be computed only for the set of selected points, and the resulting pairs ${\left\{({\mathbf{x}}_{\left[p\right]}^{\left(s\right)},{\widehat{R}}_s\left({\mathbf{x}}_{\left[p\right]}^{\left(s\right)}\right))\right\}}_{p=1}^{N_0}$ can used as compressed pattern (as $N_0\ll N$) of the system, instead of N data points ${\left\{({\mathbf{u}}_k^{\left(s\right)},{\widehat{R}}_s\left({\mathbf{u}}_k^{\left(s\right)}\right))\right\}}_{k=1}^N$.

Non-parametric pattern ${\widehat{R}}_s\left({\mathbf{x}}_{\left[1\right]}^{\left(s\right)}\right),...,{\widehat{R}}_s\left({\mathbf{x}}_{\left[N_0\right]}^{\left(s\right)}\right)$ can help to support decision of model selection from the list of potential candidates, and model competitions can be performed in the user-defined regions of interests, e.g., in the working points.