Next Article in Journal
Finite-Time Chaos Suppression of Permanent Magnet Synchronous Motor Systems
Previous Article in Journal
Co and In Doped Ni-Mn-Ga Magnetic Shape Memory Alloys: A Thorough Structural, Magnetic and Magnetocaloric Study
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

An Extended Result on the Optimal Estimation Under the Minimum Error Entropy Criterion

1
Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University, Xi'an 710049, China
2
Department of Electrical and Computer Engineering, University of Florida, Gainesville, FL 32611, USA
*
Author to whom correspondence should be addressed.
Entropy 2014, 16(4), 2223-2233; https://doi.org/10.3390/e16042223
Submission received: 24 January 2014 / Revised: 2 April 2014 / Accepted: 4 April 2014 / Published: 17 April 2014

Abstract

: The minimum error entropy (MEE) criterion has been successfully used in fields such as parameter estimation, system identification and the supervised machine learning. There is in general no explicit expression for the optimal MEE estimate unless some constraints on the conditional distribution are imposed. A recent paper has proved that if the conditional density is conditionally symmetric and unimodal (CSUM), then the optimal MEE estimate (with Shannon entropy) equals the conditional median. In this study, we extend this result to the generalized MEE estimation where the optimality criterion is the Renyi entropy or equivalently, the α-order information potential (IP).MSC Codes: 62B10

1. Introduction

Given two random variables: X ∈ ℝn which is an unknown parameter to be estimated, and Y ∈ ℝm which is the observation or measurement. The estimation of X based on Y, is in general a measurable function of Y, denoted by = g(Y) ∈ G, where G stands for the collection of all Borel measurable functions with respect to the σ -field generated by Y. The optimal estimate g* (Y) can be determined by minimizing a certain risk, which is usually a function of the error distribution. If X has conditional probability density function (PDF) p(x|y), then:

g * = arg min g G R ( p g ( x ) )

where pg (x) is the PDF of the estimate error E = Xg (Y), (.) is the risk function: E → ℝ, where E denotes the collection of all possible PDFs of the error. Let F(y) be the distribution function of Y, the PDF pg (x) will be:

p g ( x ) = m p ( x + g ( y ) y ) d F ( y )

As one can see from Equation (2), the problem of choosing an optimal g is actually the problem of shifting the components of a mixture of the conditional PDF so as to minimize the risk .

The risk function plays a central role in estimation related problems since it determines the performance surface and hence governs the optimal solution and the performance of the search algorithms. Traditional Bayes risk functions are, in general, defined as the expected value of a certain loss function (usually a nonlinear mapping) of the error:

R B a y e s ( p g ( x ) ) = n l ( x ) p g ( x ) d x

where l(.) is the loss function. The most common Bayes risk function used for estimation is the mean square error (MSE), also called the squared error or quadratic error risk, which is defined by R M S E ( p g ( x ) ) = n x 2 2 p g ( x ) d x (in this paper, ||.||p denotes the p -norm). Using the MSE as risk, the optimal estimate of X is simply the conditional mean π (y) ≜ mean [p(.|y)]. The popularity of the MSE is due to its simplicity and optimality for linear Gaussian cases [13]. However, MSE is not always a superior risk function especially for non-linear and non-Gaussian situations, since it only takes into account the second order statistics. Therefore, many alternative Bayes risk functions have been used in practical applications. The mean absolute deviation (MAD) MAD (pg (x)) = ∫n ||x||1 pg (x)dx, with which the optimal estimate is the conditional median μ (y) ≜ median [p(.|y)] (here the median of a random vector is defined as the element-wise median vector), is a robust risk function and has been successfully used in adaptive filtering in impulsive noise environments [4]. The mean 0–1 loss 0–1 (pg (x)) = ∫n l0–1 (x) pg (x) dx, where l0–1 (.)denotes the 0–1 loss function (the 0–1 loss function has been frequently used in statistics and decision theory. If the error is a discrete variable, l0–1 (x) = Entropy 16 02223f1 (x0), where Entropy 16 02223f1 (.) is the indicator function, whereas if the error is a continuous variable, l0–1 (x) is defined as l0–1 (x) = 1 – δ (x), where δ (.) is the Dirac delta function), yields the optimal estimate as ζ (y) ≜ mode [p(.|y)], i.e., the conditional mode (the mode of a continuous probability distribution is the value at which its PDF attains its maximum value), which is also the maximum a posteriori (MAP) (the MAP estimate is a limit of Bayes estimator (under the 0–1 loss function), but generally not a Bayes estimator) estimate if regarding p(.y) as the posterior density. Other important Bayes risk functions include the mean p-power error [5], Huber’s M-estimation cost [6], and the risk-sensitive cost [7], etc. For general Bayes risk Equation (3), there is no explicit expression for the optimal estimate unless some conditions on l(x) or/and conditional density p(x|y) are imposed. As shown in [8], if l(x) is even and convex, and the conditional density p(x|y) is symmetric in x, the optimal estimate will be the conditional mean (or equivalently, the conditional median).

Besides the traditional Bayes risk functions, the error entropy (EE) can also be used as a risk function in estimation problems. Using Shannon’s definition of entropy [9], the EE risk function is:

R S ( p g ( x ) ) = - n p g ( x ) log p g ( x ) d x

As the entropy measures the average dispersion or uncertainty of a random variable, its minimization makes the error concentrated. Different from conventional Bayes risks, the “loss function” of the EE risk (4) is – log pg (x), which is directly related to the error’s PDF. Therefore, when using the EE risk, we are nonlinearly transforming the error by its own PDF. In 1970, Weidemann and Stear published a paper entitled Entropy Analysis of Estimating Systems [10] in which they studied the parameter estimation problem using the error entropy as a criterion functional. They proved minimizing the error entropy is equivalent to minimizing the mutual information between error and observation, and also proved that the reduced error entropy is upper-bounded by the amount of information obtained by observation. Later, Tomita et al. [11] and Kalata and Priemer [12] studied the estimation and filtering problems from the viewpoint of information theory and derived the famed Kalman filter as a special case of minimum-error-entropy (MEE) linear estimators. Like most Bayes risks, the EE risk (4) has no explicit expression for the optimal estimate unless some constraints on the conditional density p(x|y) are imposed. In a recent paper [13], Chen and Geman proved that, if p(x|y) is conditionally symmetric and unimodal (CSUM), the MEE estimate (the optimal estimate under EE risk) will be the conditional median (or equivalently, the conditional mean or mode). Table 1 gives a summary of the optimal estimates for several risk functions. Since the entropy of a PDF remains unchanged after shift, the MEE estimator is in general restricted to be an unbiased one (i.e., with zero-mean error).

In statistical information theory, there are many extensions to Shannon’s original definition of entropy. Renyi’s entropy is one of the parametrically extended entropies. Given a random variable X with PDF p(x), α -order Renyi entropy is defined by [14]:

H α ( X ) = 1 1 - α log ( n ( p ( x ) ) α d x )

where α > 0, and α ≠ 1. The entropy definition (5) becomes the usual Shannon entropy as α → 1. Renyi entropy can be used to define a generalized EE risk:

R α ( p g ( x ) ) = 1 1 - α log ( n ( p g ( x ) ) α d x )

In recent years, the EE risk (6) has been successfully used as an adaptation cost in information theoretic learning (ITL) [1522]. It has been shown that the nonparametric kernel (Parzen window) estimator of Renyi entropy (especially when α = 2) is more computationally efficient than that of Shannon entropy [15]. The argument of the logarithm in Renyi entropy, denoted by Vα (Vα = ∫n (p(x))α dx), is called the α-order information potential (IP) (this quantity is called information potential since each term in its kernel estimator can be interpreted as a potential between two particles (see [15] for the physical interpretation of kernel estimator of information potential) [15]. As the logarithm is a monotonic function, the minimization of Renyi entropy is equivalent to the minimization (when α < 1) or maximization (when α > 1) of information potential. In practical applications, information potential has been frequently used as an alternative to Renyi entropy [15].

A natural and important question now arises: what is the optimal estimate under the generalized EE risk (6)? We do not know the answer to this question in the general case. In this work, however, we will extend the results by Chen and Geman [13] to a more general case and show that, if the conditional density p(x|y)is CSUM, the generalized MEE estimate will also be the conditional median (or equivalently, the conditional mean or mode).

2. Main Theorem and the Proof

In this section, our focus is on the α -order information potential (IP), but the conclusions drawn can be immediately transferred to Renyi entropy. The main theorem of the paper is as follows.

Theorem 1

Assume for every value y ∈ ℝm, that the conditional PDF p(x|y) is conditionally symmetric (rotation invariant for multivariate case) and unimodal (CSUM) in x ∈ ℝn, and let μ (y) = median [p(.|y)]. If α -order information potential Vα (Xμ (Y)) < ∞ (α > 0, α α 1), then:

{ V α ( X - μ ( Y ) ) V α ( X - g ( Y ) ) i f 0 < α < 1 V α ( X - μ ( Y ) ) V α ( X - g ( Y ) ) i f             α > 1

for all g :ℝm → ℝn for which Vα (Xg(Y)) < ∞.

Remark

As p(x|y) is CSUM, the conditional median μ (y) in Theorem 1 is the same as the conditional mean π (y) and conditional mode ζ (y). According to the relationship between information potential and Renyi entropy, the inequalities in Equation (7) are equivalent to:

H α ( X - μ ( Y ) ) H α ( X - g ( Y ) )

Proof of the Theorem

In this work, we give a proof for the univariate case (n = 1). A similar proof can be easily extended to the multivariate case (n > 1). In the proof we assume, without loss of generality, that ∀y, p(x|y) has median at x = 0, since otherwise we could replace p(x|y)by p(x+ μ(y)|y) and work instead with conditional densities centered at x = 0. The road map of the proof is similar to that contained in [13]. There are, however, significant differences between our work and [13]: (1) we extend the entropy minimization problem to the generalized error entropy; (2) in our proof, the Holder inequality is applied, and there is no discretization procedure, which simplifies the proof significantly.

First, we prove the following proposition:

Proposition 1

Assume that f (x|y) (not necessarily a conditional density function) satisfies

(1)

non-negative, continuous and integrable in x for each y ∈ ℝm;

(2)

symmetric (rotation invariant for n > 1) around x = 0 and unimodal for each y ∈ ℝm;

(3)

uniformly bounded in (x, y);

(4)

Vα(f0) < ∞, where Vα (f0) = ∫ (f0(x))α dx, and f0(x) = ∫m f (x|y)dF (y).

Then for all g :ℝm → ℝ for which Vα (fg) < ∞, we have

{ V α ( f 0 ) V α ( f g ) i f 0 < α < 1 V α ( f 0 ) V α ( f g ) i f             α > 1

where Vα (fg) = ∫ (fg(x))α dx, fg (x) = ∫m f (x+g (y)|y) dF (y).

Remark

It is easy to observe that ∫ f0 dx = ∫fg dx ≤ sup(x,y) f (x|y) < ∞ (not necessarily ∫ f0 dx = 1).

Proof of the Proposition

The proof is based on the following three lemmas.

Lemma 1[13]

Let non-negative function h : ℝ → [0, ∞) be bounded, continuous, and integrable, and define function Oh (z) by:

O h ( z ) = λ { x : h ( x ) z }

where λ is Lebesgue measure. Then the following results hold:

(a)

Define mh (x) = sup {z : Oh (z) ≥ x}, x ∈ (0, ∞), and mh (0) = supx h(x). Then mh (x) is continuous and non-increasing on [0, ∞), and mh (x) → 0 as x → ∞.

(b)

For any function G:[0, ∞) → ℝ with ∫ |G(h(x)) dx < ∞:

G ( h ( x ) ) d x = 0 G ( m h ( x ) ) d x

(c)

For any x0 ∈ [0, ∞):

0 x 0 m h ( x ) d x = sup A : λ ( A ) = x 0 A h ( x ) d x

Proof of Lemma 1

See the proof of Lemma 1 in [13].

Remark

The transformation hmh in Lemma1 is also called the “rearrangement” of h [23]. By Lemma 1, we have Vα (m fg) = Vα (fg) < ∞ and Vα (m f0) = Vα (f0) < ∞ (let G(x) = x α). Therefore, to prove Proposition 1, it suffices to prove:

{ V α ( m f 0 ) V α ( m f g ) i f 0 < α < 1 V α ( m f 0 ) V α ( m f g ) i f             α > 1

Lemma 2

Denote mg = mf g, m0 = mf 0. Then:

(a)

0 m g ( x ) d x = 0 m 0 ( x ) d x <

(b)

0 x 0 m g ( x ) d x 0 x 0 m 0 ( x ) d x ,             x 0 [ 0 , )

Proof of Lemma 2

See the proof of Lemma 3 in [13].

Lemma 3

α > 0, let n be a non-negative integer such that n < αn+1. Then ∀x0 ∈ [0, ∞):

(a)

0 x 0 { ( m g ( x ) ) α - n ( m 0 ( x ) ) n + 1 - α } d x 0 x 0 m 0 ( x ) d x

(b)

x 0 { ( m g ( x ) ) α - n ( m 0 ( x ) ) n + 1 - α } d x x 0 m g ( x ) d x

Proof of Lemma 3

According to Holder inequality [23], we have ∀Ω ⊂ [0, ∞):

Ω { ( m g ( x ) ) α - n ( m 0 ( x ) ) n + 1 - α } d x ( Ω m g ( x ) d x ) α - n ( Ω m 0 ( x ) d x ) n + 1 - α

By Lemma 2, 0 x 0 m g ( x ) d x 0 x 0 m 0 ( x ) d x, it follows that:

0 x 0 { ( m g ( x ) ) α - n ( m 0 ( x ) ) n + 1 - α } d x ( 0 x 0 m g ( x ) d x ) α - n ( 0 x 0 m 0 ( x ) d x ) n + 1 - α ( 0 x 0 m 0 ( x ) d x ) α - n ( 0 x 0 m 0 ( x ) d x ) n + 1 - α = 0 x 0 m 0 ( x ) d x

Further, since 0 m g ( x ) d x = 0 m 0 ( x ) d x, we have x 0 m g ( x ) d x x 0 m 0 ( x ) d x, and hence:

x 0 { ( m g ( x ) ) α - n ( m 0 ( x ) ) n + 1 - α } d x ( x 0 m g ( x ) d x ) α - n ( x 0 m 0 ( x ) d x ) n + 1 - α ( x 0 m g ( x ) d x ) α - n ( x 0 m g ( x ) d x ) n + 1 - α = x 0 m g ( x ) d x

Q.E.D. (Lemma 3)

Let Sg = sup{x : mg (x) > 0}, which is finite or infinite, Equation (17) can be rewritten as:

x 0 S g { ( m g ( x ) ) α - n ( m 0 ( x ) ) n + 1 - α } d x x 0 S g m g ( x ) d x

Now we are in position to prove Equation (13):

(1)

0 < α < 1: In this case, we have:

0 ( m g ( x ) ) α d x = 0 S g m g ( x ) ( m g ( x ) ) α - 1 d x = 0 S g m g ( x ) ( 0 ( m g ( x ) ) α - 1 d y ) d x = 0 S g { 0 m g ( x ) I ( y ( m g ( x ) ) α - 1 ) d y } d x = 0 ( inf { x : ( m g ( x ) ) α - 1 y } S g m g ( x ) d x ) d y ( A ) 0 { inf { x : ( m g ( x ) ) α - 1 y } S g ( m g ( x ) ) 1 - α ( m 0 ( x ) ) α d x } d y = 0 { 0 S g ( m g ( x ) ) 1 - α ( m 0 ( x ) ) α I ( ( m g ( x ) ) α - 1 y ) d x } d y = 0 S g ( m g ( x ) ) 1 - α ( m 0 ( x ) ) α ( 0 ( m g ( x ) ) α - 1 d y ) d x = 0 S g ( m 0 ( x ) ) α d x = ( B ) 0 S g ( m 0 ( x ) ) α d x + S g 0 ( m 0 ( x ) ) α d x = 0 ( m 0 ( x ) ) α d x

where (A) follows from Equation (21), and (B) is due to S g ( m 0 ( x ) ) α d x = 0, since

0 = S g m g ( x ) d x S g m 0 ( x ) d x 0 S g m 0 ( x ) d x = 0

(2)

α > 1: First we have

0 ( m 0 ( x ) ) α d x = 0 m 0 ( x ) ( m 0 ( x ) ) α - 1 d x = 0 m 0 ( x ) ( 0 ( m 0 ( x ) ) α - 1 d y ) d x = 0 { 0 m 0 ( x ) I ( y ( m 0 ( x ) ) α - 1 ) d y } d x = 0 ( 0 sup { x : ( m 0 ( x ) ) α - 1 y } m 0 ( x ) d x ) d y ( C ) 0 { 0 sup { x : ( m 0 ( x ) ) α - 1 y } ( m g ( x ) ) α - n ( m 0 ( x ) ) n + 1 - α d x } d y = 0 { 0 ( m g ( x ) ) α - n ( m 0 ( x ) ) n + 1 - α I ( ( m 0 ( x ) ) α - 1 y ) d x } d y = 0 ( m g ( x ) ) α - n ( m 0 ( x ) ) n + 1 - α ( 0 ( m 0 ( x ) ) α - 1 d y ) d x = 0 ( m 0 ( x ) ) n ( m g ( x ) ) α - n d x

where (C) follows from Equation (16). Further, one can derive:

0 ( m 0 ( x ) ) n ( m g ( x ) ) α - n d x = 0 ( m 0 ( x ) 0 ( m g ( x ) ) α - n ( m 0 ( x ) ) n - 1 d y ) d x = 0 { 0 m 0 ( x ) I ( y ( m g ( x ) ) α - n ( m 0 ( x ) ) n - 1 ) d y } d x = 0 { 0 sup { x : ( m g ( x ) ) α - n ( m 0 ( x ) ) n - 1 y } m 0 ( x ) d x } d y ( D ) 0 { 0 sup { x : ( m g ( x ) ) α - n ( m 0 ( x ) ) n - 1 y } m g ( x ) d x } d y = 0 { 0 m g ( x ) I ( ( m g ( x ) ) α - n ( m 0 ( x ) ) n - 1 y ) d x } d y = 0 ( m 0 ( x ) ) n - 1 ( m g ( x ) ) α - n + 1 d x 0 ( m 0 ( x ) ) n - 2 ( m g ( x ) ) α - n + 2 d x 0 ( m g ( x ) ) α d x

where (D) is because 0 x 0 m g ( x ) d x 0 x 0 m 0 ( x ) d x ,             x 0 [ 0 , ), ∀x0 ∈ [0, ∞). Combining Equations (24) and (25), we get 0 ( m 0 ( x ) ) α d x 0 ( m g ( x ) ) α d x (i.e., Vα (m0) ≥ Vα (mg)).

Up to now, the proof of Proposition 1 has been completed. Let us come back to the proof of Theorem 1. The remaining task is just to remove the conditions of continuity and uniform boundedness imposed in Proposition 1. This can be easily accomplished by approximating p(x|y) by a sequence of functions {fn (x|y)}, n = 1, 2,..., which satisfy the conditions of Proposition 1. Then similar to [13], we define:

f n ( x y ) = { n x x + ( 1 / n ) min ( n , p ( z y ) ) d z             x [ 0 , ) f n ( - x y )             x ( - , 0 )

It is easy to verify that for each n, fn (x|y) satisfies all the conditions of Proposition 1. Here we only give the proof for condition 4). Let f n 0 ( x ) = m f n ( x y ) d F ( y ), we have:

V α ( f n 0 ) = ( f n 0 ( x ) ) α d x = ( m f n ( x y ) d F ( y ) ) α d x = 2 + ( m ( n x x + ( 1 / n ) min ( n , p ( z y ) ) d z ) d F ( y ) ) α d x 2 + ( m ( n x x + ( 1 / n ) sup z [ x , x + ( 1 / n ) ] p ( z y ) d z ) d F ( y ) ) α d x = ( E ) 2 + ( m ( n x x + ( 1 / n ) p ( x y ) d z ) d F ( y ) ) α d x = 2 + ( m p ( x y ) d F ( y ) ) α d x = V α ( p 0 ) <

where (E) comes from the fact that ∀y, p(x|y)is non-increasing in x over [0, ∞), since it is CSUM.

According to Proposition 1, we have, for every n :

{ V α ( f n 0 ) V α ( f n g ) i f 0 < α < 1 V α ( f n 0 ) V α ( f n g ) i f             α > 1

where f n g ( x ) m f n ( x + g ( y ) y ) d F ( y ). In order to complete the proof of Theorem 1, we only need to show that V α ( f n 0 ) V α ( p 0 ), and V α ( f n g ) V α ( p g ). This can be proved by the dominated convergence theorem. Here we only show V α ( f n 0 ) V α ( p 0 ), the proof for V α ( f n g ) V α ( p g ) is identical. First, it is clear that f n 0 ( x ) p 0 ( x ), ∀x, and hence ( f n 0 ( x ) ) α ( p 0 ( x ) ) α, ∀x. Also, we can derive:

f n 0 ( x ) - p 0 ( x ) d x 0

As Vα (p0) = ∫ (p0(x))α < ∞, then by dominated convergence theorem, V α ( f n 0 ) V α ( p 0 ).

Q.E.D. (Theorem 1)

Remark

The condition of the CSUM in Theorem 1 is a little strong, but it can be easily relaxed to just requiring that the conditional PDF p(x|y) is generalized uniformly dominated (GUD) in x ∈ ℝn (see [24] for the definition of GUD).

3. Conclusions

The problem of determining a minimum-error-entropy (MEE) estimator is actually the problem of shifting the components of a mixture of the conditional PDF so as to minimize the entropy of the mixture. It has been proved in a recent paper that, if the conditional distribution is conditionally symmetric and unimodal (CSUM), the Shannon entropy of the mixture distribution will be minimized by aligning the conditional median. In the present work, this result has been extended to a more general case. We show that if the conditional distribution is CSUM, the Renyi entropy of the mixture distribution will also be minimized by aligning the conditional median.

Acknowledgments

This work was supported by National Natural Science Foundation of China (No. 61372152, No. 90920301) and 973 Program (No.2012CB316400, No. 2012CB316402).

Conflicts of Interest

The authors declare no conflict of interest.

  • Author ContributionsThe contributions of each author are as follows: Badong Chen proved the main theorem and finished the draft; Guangmin Wang polished the language and typeset the manuscript; Nanning Zheng was in charge of technical checking; Jose C. Principe proofread the paper.

References

  1. Haykin, S. Adaptive Filtering Theory; Prentice Hall: New York, NY, USA, 1996. [Google Scholar]
  2. Kailath, T.; Sayed, A.H.; Hassibi, B. Linear Estimation; Prentice Hall: Englewood Cliffs, NJ, USA, 2000. [Google Scholar]
  3. Papoulis, A.; Pillai, S.U. Probability, Random Variables, and Stochastic Processes; McGraw-Hill Education: New York, NY, USA, 2002. [Google Scholar]
  4. Shao, M.; Nikias, C.L. Signal processing with fractional lower order moments: Stable processes and their applications. IEEE Proc 1993, 81, 986–1009. [Google Scholar]
  5. Pei, S.C.; Tseng, C.C. Least mean p-power error criterion for adaptive FIR filter. IEEE J. Sel. Areas Commun 1994, 12, 1540–1547. [Google Scholar]
  6. Rousseeuw, P.J.; Leroy, A.M. Robust Regression and Outlier Detection; John Wiley & Sons: New York, NY, USA, 1987. [Google Scholar]
  7. Boel, R.K.; James, M.R.; Petersen, I.R. Robustness and risk-sensitive filtering. IEEE Trans. Automat. Control 2002, 47, 451–461. [Google Scholar]
  8. Hall, E.B.; Wise, G.L. On optimal estimation with respect to a large family of cost functions. IEEE Trans. Inform. Theory 1991, 37, 691–693. [Google Scholar]
  9. Cover, T.M.; Thomas, J.A. Element of Information Theory; Wiley & Son, Inc: New York, NY, USA, 1991. [Google Scholar]
  10. Weidemann, H.L.; Stear, E.B. Entropy analysis of estimating systems. IEEE Trans. Inform. Theory 1970, 16, 264–270. [Google Scholar]
  11. Tomita, Y.; Ohmatsu, S.; Soeda, T. An application of the information theory to estimation problems. Inf. Control 1976, 32, 101–111. [Google Scholar]
  12. Kalata, P.; Priemer, R. Linear prediction, filtering and smoothing: An information theoretic approach. Inf. Sci 1979, 17, 1–14. [Google Scholar]
  13. Chen, T.-L.; Geman, S. On the minimum entropy of a mixture of unimodal and symmetric distributions. IEEE Trans. Inf. Theory 2008, 54, 3166–3174. [Google Scholar]
  14. Renyi, A. On Measures of Entropy and Information. Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Statistical Laboratory of the University of California, Berkeley, CA, USA, 20 June–30 July, 1960; 1, pp. 547–561.
  15. Principe, J.C. Information Theoretic Learning: Renyi’s Entropy and Kernel Perspectives; Springer: New York, NY, USA, 2010. [Google Scholar]
  16. Chen, B.; Zhu, Y.; Hu, J.; Principe, J.C. System Parameter Identification: Information Criteria and Algorithms; Elsevier Inc: London, UK, 2013. [Google Scholar]
  17. Erdogmus, D.; Principe, J.C. An error-entropy minimization algorithm for supervised training of nonlinear adaptive systems. IEEE Trans. Signal Process 2002, 50, 1780–1786. [Google Scholar]
  18. Erdogmus, D.; Principe, J.C. Generalized information potential criterion for adaptive system training. IEEE Trans. Neur. Netw 2002, 13, 1035–1044. [Google Scholar]
  19. Erdogmus, D.; Principe, J.C. From linear adaptive filtering to nonlinear information processing—The design and analysis of information processing systems. IEEE Signal Process. Mag 2006, 23, 14–33. [Google Scholar]
  20. Santamaria, I.; Erdogmus, D.; Principe, J.C. Entropy minimization for supervised digital communications channel equalization. IEEE Trans. Signal Process 2002, 50, 1184–1192. [Google Scholar]
  21. Chen, B.; Hu, J.; Pu, L.; Sun, Z. Stochastic gradient algorithm under (h, φ)-entropy criterion. Circuits Syst. Signal Process 2007, 26, 941–960. [Google Scholar]
  22. Chen, B.; Zhu, Y.; Hu, J. Mean-square convergence analysis of ADALINE training with minimum error entropy criterion. IEEE Trans. Neur. Netw 2010, 21, 1168–1179. [Google Scholar]
  23. Hardy, G.H.; Littlewood, J.E.; Polya, G. Inequalities; Cambridge University Press: Cambridge, UK, 1934. [Google Scholar]
  24. Chen, B.; Principe, J.C. Some further results on the minimum error entropy estimation. Entropy 2012, 14, 966–977. [Google Scholar]
Table 1. Optimal estimates for several risk functions.
Table 1. Optimal estimates for several risk functions.
Risk function (pg(x))Optimal estimate
Mean square error (MSE) n x 2 2 p g ( x ) d xg* (y) = π(y) ≜ mean [p(.|y)]
Mean absolute deviation (MAD)n ||x||1 pg (x) dxg* (y) = μ(y) ≜ median [p(.|y)]
Mean 0–1 lossn l0–1 (x) pg (x)dxg* (y) = ζ (y) ≜ mode [p(.|y)]
General Bayes riskn l (x) pg (x) dxIf l(x) is even and convex, and p(x|y) is symmetric in x, then g* (y) = π(y) = μ(y)
Error entropy (EE)−∫n pg (x) log pg (x) dxIf p(x|y) is CSUM, then g*(y) = π(y) = μ(y) = ζ(y)

Share and Cite

MDPI and ACS Style

Chen, B.; Wang, G.; Zheng, N.; Principe, J.C. An Extended Result on the Optimal Estimation Under the Minimum Error Entropy Criterion. Entropy 2014, 16, 2223-2233. https://doi.org/10.3390/e16042223

AMA Style

Chen B, Wang G, Zheng N, Principe JC. An Extended Result on the Optimal Estimation Under the Minimum Error Entropy Criterion. Entropy. 2014; 16(4):2223-2233. https://doi.org/10.3390/e16042223

Chicago/Turabian Style

Chen, Badong, Guangmin Wang, Nanning Zheng, and Jose C. Principe. 2014. "An Extended Result on the Optimal Estimation Under the Minimum Error Entropy Criterion" Entropy 16, no. 4: 2223-2233. https://doi.org/10.3390/e16042223

Article Metrics

Back to TopTop