Next Article in Journal
Entropy Generation and Heat Transfer Performance in Microchannel Cooling
Next Article in Special Issue
An Information Criterion for Auxiliary Variable Selection in Incomplete Data Analysis
Previous Article in Journal
Nonrigid Medical Image Registration Using an Information Theoretic Measure Based on Arimoto Entropy with Gradient Distributions
Previous Article in Special Issue
The Optimized Multi-Scale Permutation Entropy and Its Application in Compound Fault Diagnosis of Rotating Machinery
Article Menu
Issue 2 (February) cover image

Export Article

Open AccessArticle
Entropy 2019, 21(2), 190; https://doi.org/10.3390/e21020190

Mixture of Experts with Entropic Regularization for Data Classification

1
Department of Engineering Science, Andres Bello University, Santiago 7500971, Chile
2
Department of Engineering Informatics, Catholic University of Temuco, Temuco 4781312, Chile
3
Department of Computer Sciences, Pontifical Catholic University of Chile, Santiago 7820436, Chile
*
Author to whom correspondence should be addressed.
Received: 4 January 2019 / Revised: 4 February 2019 / Accepted: 15 February 2019 / Published: 18 February 2019
(This article belongs to the Special Issue Information-Theoretical Methods in Data Mining)
Full-Text   |   PDF [3604 KB, uploaded 18 February 2019]   |  

Abstract

Today, there is growing interest in the automatic classification of a variety of tasks, such as weather forecasting, product recommendations, intrusion detection, and people recognition. “Mixture-of-experts” is a well-known classification technique; it is a probabilistic model consisting of local expert classifiers weighted by a gate network that is typically based on softmax functions, combined with learnable complex patterns in data. In this scheme, one data point is influenced by only one expert; as a result, the training process can be misguided in real datasets for which complex data need to be explained by multiple experts. In this work, we propose a variant of the regular mixture-of-experts model. In the proposed model, the cost classification is penalized by the Shannon entropy of the gating network in order to avoid a “winner-takes-all” output for the gating network. Experiments show the advantage of our approach using several real datasets, with improvements in mean accuracy of 3–6% in some datasets. In future work, we plan to embed feature selection into this model. View Full-Text
Keywords: mixture-of-experts; regularization; entropy; classification mixture-of-experts; regularization; entropy; classification
Figures

Figure 1

This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).
SciFeed

Share & Cite This Article

MDPI and ACS Style

Peralta, B.; Saavedra, A.; Caro, L.; Soto, A. Mixture of Experts with Entropic Regularization for Data Classification. Entropy 2019, 21, 190.

Show more citation formats Show less citations formats

Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Related Articles

Article Metrics

Article Access Statistics

1

Comments

[Return to top]
Entropy EISSN 1099-4300 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top