Next Article in Journal
On Using Entropy for Enhancing Handwriting Preprocessing
Previous Article in Journal
Viscosity in Modified Gravity
Article Menu

Export Article

Open AccessArticle
Entropy 2012, 14(11), 2311-2323;

On the Smoothed Minimum Error Entropy Criterion

Department of Electrical and Computer Engineering, University of Florida, Gainesville, FL 32611, USA
Department of Precision Instruments and Mechanology, Tsinghua University, Beijing, 100084, China
Author to whom correspondence should be addressed.
Received: 9 July 2012 / Revised: 1 November 2012 / Accepted: 1 November 2012 / Published: 12 November 2012
Full-Text   |   PDF [228 KB, uploaded 24 February 2015]   |  


Recent studies suggest that the minimum error entropy (MEE) criterion can outperform the traditional mean square error criterion in supervised machine learning, especially in nonlinear and non-Gaussian situations. In practice, however, one has to estimate the error entropy from the samples since in general the analytical evaluation of error entropy is not possible. By the Parzen windowing approach, the estimated error entropy converges asymptotically to the entropy of the error plus an independent random variable whose probability density function (PDF) corresponds to the kernel function in the Parzen method. This quantity of entropy is called the smoothed error entropy, and the corresponding optimality criterion is named the smoothed MEE (SMEE) criterion. In this paper, we study theoretically the SMEE criterion in supervised machine learning where the learning machine is assumed to be nonparametric and universal. Some basic properties are presented. In particular, we show that when the smoothing factor is very small, the smoothed error entropy equals approximately the true error entropy plus a scaled version of the Fisher information of error. We also investigate how the smoothing factor affects the optimal solution. In some special situations, the optimal solution under the SMEE criterion does not change with increasing smoothing factor. In general cases, when the smoothing factor tends to infinity, minimizing the smoothed error entropy will be approximately equivalent to minimizing error variance, regardless of the conditional PDF and the kernel. View Full-Text
Keywords: entropy; supervised machine learning; minimum error entropy criterion entropy; supervised machine learning; minimum error entropy criterion

Figure 1

This is an open access article distributed under the Creative Commons Attribution License (CC BY 3.0).

Share & Cite This Article

MDPI and ACS Style

Chen, B.; Principe, J.C. On the Smoothed Minimum Error Entropy Criterion. Entropy 2012, 14, 2311-2323.

Show more citation formats Show less citations formats

Related Articles

Article Metrics

Article Access Statistics



[Return to top]
Entropy EISSN 1099-4300 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top