Special Issue "Information–Theoretic Approaches to Computational Intelligence"

A special issue of Entropy (ISSN 1099-4300). This special issue belongs to the section "Information Theory, Probability and Statistics".

Deadline for manuscript submissions: 20 November 2019.

Special Issue Editors

Prof. Dr. Gernot Kubin
E-Mail Website
Guest Editor
Signal Processing and Speech Communication Laboratory, Graz University of Technology, Graz, Austria
Interests: computational intelligence; signal processing; speech communication
Dr. Bernhard C. Geiger
E-Mail Website
Guest Editor
Know-Center GmbH, Graz, Styria, Austria
Interests: information theory for machine learning and signal processing; information–theoretic model reduction; stochastic processes

Special Issue Information

Dear Colleagues,

Computational Intelligence (CI) concerns information processing with biologically or linguistically inspired methods, such as neural networks, evolutionary computing, fuzzy logic, probabilistic and Bayesian methods, and learning theory. In contrast to this method-centered definition of CI, information theory concerns the fundamental limits of information processing, irrespective of the employed method. Nevertheless, information theory offers a great selection of quantities capturing statistical dependencies and similarities that have found applications as diverse as the analysis of cognitive processes and the design of man-made systems.

An information–theoretic approach to CI shall, therefore, concern the theory-driven development or theoretical analysis of CI methods with information–theoretic quantities. Specifically, such an approach shall give answers to the “what”, “how”, and “why”. What happens to a piece of information that is processed by a CI method? How is this information processing achieved? Why should information be processed in exactly this way? In this Special Issue, we hence seek contributions on:

- Input-to-output behavior of CI methods (e.g., the effect of a CI method on the information content of the processed signal). Theoretical or experimental analyses, using information–theoretic quantities, are welcome.

- Dynamic behavior of CI methods (e.g., information propagation in evolutionary computing, information–theoretic aspects of neural network training). Theoretical or experimental analyses, using information–theoretic quantities, are welcome.

- Development of CI methods using information–theoretic cost functions (e.g., information-maximizing CI methods, CI methods based on rate-distortion theory). Application-specific experimental analyses of information–theoretic cost functions are welcome, given that the proposed cost functions are justified from first principles and rigorously developed.

We wish to mention that the concurrent Special Issue “Information–Theoretic Approaches in Deep Learning” has a potential overlap with ours. We therefore reserve the right to prescreen submissions focused on neural networks and forward them to this other Special Issue in case of a better fit.

Prof. Dr. Gernot Kubin
Dr. Bernhard C. Geiger
Guest Editors

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All papers will be peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Entropy is an international peer-reviewed open access monthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 1600 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

Keywords

  • evolutionary computing
  • neural networks
  • probabilistic methods
  • fuzzy logic
  • learning theory
  • information–theoretic cost functions
  • input–output behavior

Published Papers (3 papers)

Order results
Result details
Select all
Export citation of selected articles as:

Research

Open AccessArticle
Markov Information Bottleneck to Improve Information Flow in Stochastic Neural Networks
Entropy 2019, 21(10), 976; https://doi.org/10.3390/e21100976 - 06 Oct 2019
Abstract
While rate distortion theory compresses data under a distortion constraint, information bottleneck (IB) generalizes rate distortion theory to learning problems by replacing a distortion constraint with a constraint of relevant information. In this work, we further extend IB to multiple Markov bottlenecks (i.e., [...] Read more.
While rate distortion theory compresses data under a distortion constraint, information bottleneck (IB) generalizes rate distortion theory to learning problems by replacing a distortion constraint with a constraint of relevant information. In this work, we further extend IB to multiple Markov bottlenecks (i.e., latent variables that form a Markov chain), namely Markov information bottleneck (MIB), which particularly fits better in the context of stochastic neural networks (SNNs) than the original IB. We show that Markov bottlenecks cannot simultaneously achieve their information optimality in a non-collapse MIB, and thus devise an optimality compromise. With MIB, we take the novel perspective that each layer of an SNN is a bottleneck whose learning goal is to encode relevant information in a compressed form from the data. The inference from a hidden layer to the output layer is then interpreted as a variational approximation to the layer’s decoding of relevant information in the MIB. As a consequence of this perspective, the maximum likelihood estimate (MLE) principle in the context of SNNs becomes a special case of the variational MIB. We show that, compared to MLE, the variational MIB can encourage better information flow in SNNs in both principle and practice, and empirically improve performance in classification, adversarial robustness, and multi-modal learning in MNIST. Full article
(This article belongs to the Special Issue Information–Theoretic Approaches to Computational Intelligence)
Open AccessArticle
Learnability for the Information Bottleneck
Entropy 2019, 21(10), 924; https://doi.org/10.3390/e21100924 - 23 Sep 2019
Abstract
The Information Bottleneck (IB) method provides an insightful and principled approach for balancing compression and prediction for representation learning. The IB objective I ( X ; Z ) β I ( Y ; Z ) employs a Lagrange multiplier β to tune [...] Read more.
The Information Bottleneck (IB) method provides an insightful and principled approach for balancing compression and prediction for representation learning. The IB objective I ( X ; Z ) β I ( Y ; Z ) employs a Lagrange multiplier β to tune this trade-off. However, in practice, not only is β chosen empirically without theoretical guidance, there is also a lack of theoretical understanding between β , learnability, the intrinsic nature of the dataset and model capacity. In this paper, we show that if β is improperly chosen, learning cannot happen—the trivial representation P ( Z | X ) = P ( Z ) becomes the global minimum of the IB objective. We show how this can be avoided, by identifying a sharp phase transition between the unlearnable and the learnable which arises as β is varied. This phase transition defines the concept of IB-Learnability. We prove several sufficient conditions for IB-Learnability, which provides theoretical guidance for choosing a good β . We further show that IB-learnability is determined by the largest confident, typical and imbalanced subset of the examples (the conspicuous subset), and discuss its relation with model capacity. We give practical algorithms to estimate the minimum β for a given dataset. We also empirically demonstrate our theoretical conditions with analyses of synthetic datasets, MNIST and CIFAR10. Full article
(This article belongs to the Special Issue Information–Theoretic Approaches to Computational Intelligence)
Show Figures

Figure 1

Open AccessArticle
Gaussian Mean Field Regularizes by Limiting Learned Information
Entropy 2019, 21(8), 758; https://doi.org/10.3390/e21080758 - 03 Aug 2019
Cited by 1
Abstract
Variational inference with a factorized Gaussian posterior estimate is a widely-used approach for learning parameters and hidden variables. Empirically, a regularizing effect can be observed that is poorly understood. In this work, we show how mean field inference improves generalization by limiting mutual [...] Read more.
Variational inference with a factorized Gaussian posterior estimate is a widely-used approach for learning parameters and hidden variables. Empirically, a regularizing effect can be observed that is poorly understood. In this work, we show how mean field inference improves generalization by limiting mutual information between learned parameters and the data through noise. We quantify a maximum capacity when the posterior variance is either fixed or learned and connect it to generalization error, even when the KL-divergence in the objective is scaled by a constant. Our experiments suggest that bounding information between parameters and data effectively regularizes neural networks on both supervised and unsupervised tasks. Full article
(This article belongs to the Special Issue Information–Theoretic Approaches to Computational Intelligence)
Show Figures

Figure 1

Planned Papers

The below list represents only planned manuscripts. Some of these manuscripts have not been received by the Editorial Office yet. Papers submitted to MDPI journals are subject to peer-review.

Title: Information Theoretic Concept Drift Detector
Authors: Shujian Yu, Francisco J. Valverde-Albacete, Carmen Pelaez-Moreno, Jose C. Principe
Abstract: Error-based methods have been widely used to detect concept drifts (i.e., changes in the joint distribution between predictor and response variables) in streaming data. Hence, the selection of the adopted error-related statistic is crucial to the performance of different concept drift detectors. In this paper, we suggest three novel error-related statistics in the entropy triangle, namely the redundancy, the mutual information and the variation of information, and use the Hoeffding’s inequality to monitor these three statistics over a sliding window. Experimental results, on both synthetic and real data, suggest that our new statistics perform well across different concept drift types (e.g., gradual or abrupt, recurrent or irregular) and different data stream distributions (e.g., balanced and imbalanced labels), regardless of the number of categories.

Back to TopTop