Entropy and Information Theory in Machine Learning: Theoretical Insights and Applications

Dear Colleagues,

This Special Issue of Entropy titled, “Entropy and Information Theory in Machine Learning: Theoretical Insights and Applications“, is a second-volume sequel to our first Special Issue titled, “Adaptive signal processing and Machine Learning Using Entropy and Information Theory“.

Adaptive signal processing, machine learning and deep learning, which rely on the paradigm of learning from data, have become indispensable tools for extracting information, making decisions and interacting with our environment. The information extraction process is a very critical step in this process. Many of the algorithms deployed for information extraction have largely been based on using the popular mean square error (MSE) criterion. They leverage the significant information contained in the data. The more accurate the process of extracting useful information from the data, the more precise and efficient the learning and signal processing will be.

It is well-known that information theoretic learning (ITL)-based cost measures can provide better nonlinear models in a range of problems from system identification and regression to classification. Information theoretic learning (ITL) has initially been applied for such supervised learning applications. ITL-based cost measures also perform better when the error distribution is non-Gaussian, such as in supervised learning.

Entropy and information theory have always represented useful tools to deal with information and the amount of information contained in a random variable. Information theory mainly relies on the basic intuition that learning that an unlikely event has occurred is more informative than learning that a likely event has occurred. Entropy gives a measure of the amount of information in an event drawn from a distribution. For this reason, they have been widely used in adaptive signal processing and machine learning to improve performance by designing and optimizing effective and specific models that fit the data, even in noisy and adverse scenario conditions.

The presence of strong disturbances in the error signal can severely deteriorate the convergence behavior of adaptive filters and, in some cases, cause the learning algorithms to diverge. Information theoretic learning (ITL) approaches have recently emerged as an effective solution to handle such scenarios.

Examples of several measures widely adopted include mutual information, cross-entropy, minimum error entropy (MEE) criterion, maximum correntropy criterion (MCC) and Kullback–Leibler divergence, among others. Moreover, a wide class of interesting tasks of adaptive signal processing, machine learning and deep learning take advantage of entropy and information theory, including: exploratory data analysis, feature and model selection, sampling and subset extraction, optimizing learning algorithms, clustering sensitivity analysis, representation learning, and data generation.

This Special Issue aims at providing an avenue for the publication of recent developments in the areas of entropy and information theoretic-based measures used in machine learning. We solicit papers expounding on theoretical insights as well as the latest applications of these techniques for solving various problems.

Prof. Dr. Tokunbo Ogunfunmi
Dr. Nithin V. George
Dr. Danilo Comminiello
Guest Editors

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 250 words) can be sent to the Editorial Office for assessment.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-anonymized peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Entropy is an international peer-reviewed open access monthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 2600 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.