This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).
The paper presents a new approach to restoration characteristics randomized models under small amounts of input and output data. This approach proceeds from involving randomized static and dynamic models and estimating the probabilistic characteristics of their parameters. We consider static and dynamic models described by Volterra polynomials. The procedures of robust parametric and nonparametric estimation are constructed by exploiting the entropy concept based on the generalized informational Boltzmann’s and Fermi’s entropies.
The problem of useful information retrieval (we comprehend it as parametric and nonparametric estimation based on real data) is a major one in modern science. Different scientific disciplines suggest numerous methods of solving this problem. Each method stems from certain hypotheses regarding the properties of data accumulated during the normal functioning of their source. Among advanced scientific disciplines in this field, we mention mathematical statistics [
Methods developed within their frameworks rest upon two groups of fundamental hypotheses. The first one relates to
The second group of hypotheses applies to data and plays an essential role. In fact, these hypotheses are stated in terms of the statistical properties of data arrays (e.g., a sufficient number of data arrays, a property of a sample from a universal set, normal probability density). In practice, it seems impossible to check such properties and assumptions in concrete problems.
The described situation happens in a class of problems, where the sizes of real data arrays are limited and data incorporate errors [
Consequently, the characteristics (parameters) of a model are estimated by a small amount of incompletely reliable data. We can treat them as random objects. In this case, the estimated characteristics of a model acquire the properties of random variables.
Therefore, one naturally arrives at the idea of considering model parameters as random quantities. This idea transforms the model with deterministic parameters to the model with random parameters. In the sequel, we adopt the term of a randomized model (RM). The characteristics of an RM include the probability density functions (pdfs) of the model parameters. Thus, one should estimate the pdfs of model parameters (not the estimates of their values) based on available data. Having such estimates at one’s disposal, one can apply an RM for:
constructing = moment models (MMs), where appropriate moments of random parameters serve as the model parameters;
generating an ensemble of random vectors (ERV) of RM “output” with a pdf estimate (by the Monte Carlo method) and performing the statistical processing of the ensemble to form the desired numerical characteristics (including moment ones).
Both directions of RM usage enlarge appreciably the application domains of such models (especially, the ones with a high level of uncertainties). However, a researcher still faces a certain problem. How could the probability density functions of parameters be estimated in randomized models?
This paper proposes involving the informational entropy maximization principle on sets defined by the “inputoutput” measurements of an RM. The proposal originates from a couple of considerations discussed and formalized below.
The first consideration is connected with the generalized notion of likelihood,
The second consideration is based on methodological interpretations of the notion of informational entropy as a measure of uncertainty. Entropy maximization guarantees the best solutions under the maximal uncertainty. This line of reasoning was pioneered in [
Consider a static parameterized object with a measurable “input” described by a matrix,
The relationship between the “input” and “output” (including measurement errors) is defined by the randomized static model (RSM):
We adopt the following notation:
The linear modification of the RSM acquires the form:
Measurements incorporate errors modeled by a matrix,
Consider a discrete dynamic object with a finite “memory”,
The connection between the observed input and output (including output measurement errors) is defined by a randomized dynamic model (RDM). This model is described by the discrete functional Volterra polynomial of degree,
Equality
The discrete dynamic model has the nonlinear expression
According to the accepted numbering, adopt the following indexing of random parameters that correspond to the values of weight functions in
Construct the random vector:
Components of the vector,
By virtue of
Similarly to
Consequently, for fixed
Therefore, equality
Consider the interval
Define the input measurement matrices, whose rows are formed from vector
Build the block matrix,
Finally, construct the block random vector of model parameters (of length
Thus, dynamic model
Structurally, this expression is analogous to linear static data model
Numerical characteristics of RMs are understood as the probability density functions (alternatively, probability functions) of model parameters and noise components.
In the sequel, we believe that RMs whose random components are described by probability density functions belong to the class RMPWQ. By analogy, RMs described by the probabilities of lying within appropriate intervals will form the class RMpwq.
The parameters of an RDM from this class and the measurement noises represent continuous random variables. They take values in intervals, where there exist probability density functions (pdfs):
for the random parameters of an RSM:
 for the random parameters of an RDM:
 for the “input” measurement noises:
 for the “output” measurement noises:
The abovementioned pdfs of the random parameters and noises should be estimated using measurements of the RM “input” and “output” and
An RM generates an ensemble,
This paper utilizes the first moments,
For the dynamic model RMPQ
The parameters of an RM from this class and the measurement noises turn out continuous random variables. Their
The parameters
Just like models from the class RMPWQ, RMs belonging to the class considered to reproduce an ensemble,
As a numerical characteristic for the ensemble generated by an RM of this class, select the vector of the first
In the previous formula, ⊗ stands for elementwise multiplication,
The transform
By applying
In the case of RDMs, the first quasimoment formula of the random vector,
Concluding this section, we emphasize a relevant aspect. For the class RMPWQ, it is necessary to estimate probability density functions. For the class RMpwq, one has to estimate vectors characterizing probability distributions.
We propose to introduce a
Suppose that we know the
Following [
Hence, the LLR represents a nonrandom function of random arguments.
The probabilities,
Now, introduce the likelihood functional (the generalized informational Boltzmann entropy functional),
Obviously (see
Define the likelihood function (the generalized informational Boltzmann entropy function) by (see [
The estimation quality for pdfs or probability vector components is characterized by the maximal value of the generalized informational Boltzmann entropy functional (or function). In the sequel, we distinguish between two types of estimates.
The
the probability density functions belong to the class:
the balance condition holds true for the degree (1
The
the probability distributions belong to the class:
the balance condition holds true for the quasiaverage vector,
Consider the subclass of powertype RSMPQ with the following nonlinearities:
The vector of parameters,
The “input” and “output” are observed at instants
Additionally, the observed “output” of the RM is characterized by the random vector
Therefore, using observation results, we rewrite the RSMPQ as:
Here,
To proceed, we analyze the
Revert to problem
The structural properties of the
In the last formulas, the Lagrange multipliers,
For nonlinear RMs, the structure of the
Address the linear form of RMPQ
The vector, Φ̄[
Problem
Here,
Consider the subclass of linear RSMpq with accurate “input” measurements:
Denote by
Let us study the
This problem belongs to the class of entropylinear programming problems [
Consequently, we derive a system of equations determining the entropyoptimal probabilities,
Lagrange’s method of multipliers gives the following system of equations for the estimates of the probabilities,
So long as the estimates of
Under certain conditions, the estimates generated by Problems 1–2 differ in the values of some entropy (e.g., generalized Boltzmann entropy
the entropy:
the estimate,
the absolute maximum of the entropy
the set:
These distances coincide only if
Consider a linear RM with five random parameter
The reference model has fixed parameters
The “input” measurement matrix makes up:
The “output” measurement vector (distorted by noises from appropriate intervals) takes the form:
The first quasimoments for the parameters and noise are defined by:
Substituting the last equalities into
As
Scenario
We study the
The entropy,
Solutions to Problem 1 can be found in
Solutions to Problem 2 are shown by
The comparative analysis of these computations draws the following conclusions:
the estimates of the parameters derived for interval probabilities have a larger constrained maximum of the entropy than the ones derived for normalized probabilities (see Theorem 2);
the reference parameters and
Consider the following RDMPQ:
Therefore, the dynamic RMPQ in question incorporates nine parameters:
The values of constants in
We have two measurements of the “input” and “output.” Construct the blocks
The matrix
The noise components become
The
The Lagrange multipliers,
To solve these equations, we have used MATLAB for symbolic transformations and the numerical solution of nonlinear equations, known as the “trustregion dogleg” technique. The computed Lagrange multipliers form
New methods of parametric (probabilities) and nonparametric (probability density functions) estimation of the randomized model characteristics are proposed. These methods are based on entropy functions or entropy functionals maximized under certain constraints. We can interpret the obtained estimate as a robust one, as the entropy function was used for its calculation. These methods are focused on the problems where data is limited and distorted by noises. It is shown that the entropyrobust estimation of the probabilities and probability density functions belong to the exponential class.
The authors declare no conflict of interest.
PDFs of the parameters.
PDFs of the noise.
The
1.0  1.0  1.0  1.0  1.0  
0.1  0.2  0.3  0.3  0.1  
0.3  0.4  0.1  0.05  0.15 
The
0.2  0.8  
1.0  1.0 
The components,
0.36  0.36  0.36  0.36  0.36  
0.036  0.072  0.108  0.108  0.036  
0.108  0.144  0.036  0.018  0.054 
The components,
0.072  0.288  
0.36  0.36 
The









0.14  0.17  0.28  0.20  0.21  0.28  0.72  1.56  
0.08  0.16  0.26  0.34  0.16  0.25  0.75  −0.03  
0.17  0.31  0.22  0.06  0.24  0.40  0.60  −0.23  
0.23  0.22  0.19  0.19  0.16  0.45  0.55  2.29  
0.14  0.24  0.18  0.32  0.12  0.39  0.61  0.63  
0.26  0.0.38  0.13  0.05  0.18  0.58  0.42  0.67 
The first quasimoments of the parameters and noises in Problem 1.









1.43  1.66  2.78  2.01  2.13  −1.33  2.65  0.13  
0.78  1.64  2.62  3.37  1.58  −1.52  3.04  0.03  
1.67  3.08  2.25  0.63  2.37  −0.62  1.23  0.30  
2.35  2.22  1.91  1.88  1.64  −0.31  0.62  0.15  
1.41  2.36  1.84  3.21  1.18  −0.65  1.30  0.02  
2.56  3.76  1.30  0.56  1.82  0.46  −0.91  0.36 
The









0.24  0.22  0.18  0.23  0.26  0.0.07  0.0.29  2.05  
0.15  0.26  0.27  0.27  0.07  0.07  0.46  0.27  
0.23  0.39  0.24  0.05  0.12  0.11  0.26  0.23  
0.25  0.22  0.15  0.22  0.25  0.30  0.39  2.37  
0.15  0.26  0.23  0.26  0.06  0.34  0.60  0.67  
0.26  0.40  0.17  0.05  0.11  0.48  0.37  0.71 
The 1̃quasimoments of the parameters and noise in Problem 2.









2.37  2.26  1.83  2.35  2.64  −2.61  −2.57  0.14  
1.50  2.64  2.71  2.74  0.68  −2.58  −0.47  0.06  
2.34  3.95  2.44  0.52  1.23  −2.34  −2.92  0.33  
2.50  2.24  1.50  2.21  2.50  −1.17  −1.35  0.16  
1.52  2.58  2.35  2.60  0.66  −0.97  1.21  0.06  
2.62  3.98  1.68  0.47  1.11  −0.12  −1.51  0.36 
The values of









1.0  2.0  0.5  1.0  0.08  0.08  0.08  0.08 
The intervals for the parameters.

0.50  0.46  0.42  1.00  0.92  0.85  0.85  0.79  0.72 

1.00  0.92  0.85  2.00  1.84  1.70  1.70  1.58  1.44 
The Lagrange multipliers,
−10.0475  −6.8421  −13.7688  1.5448  11.2746  −37.2191 
The Lagrange multipliers,
−32.9907  −15.0797  −29.7189 
The Lagrange multipliers,
26.6458  14.7240  
9.9822  −2.7918 