Next Article in Journal
Asymptotic Analysis of the kth Subword Complexity
Previous Article in Journal
Rope Tension Fault Diagnosis in Hoisting Systems Based on Vibration Signals Using EEMD, Improved Permutation Entropy, and PSO-SVM
Previous Article in Special Issue
Mining Educational Data to Predict Students’ Performance through Procrastination Behavior
Open AccessArticle

Nonlinear Canonical Correlation Analysis:A Compressed Representation Approach

1
The Industrial Engineering Department, Tel Aviv University, Tel Aviv 6997801, Israel
2
The School of Electrical Engineering, Tel Aviv University, Tel Aviv 6997801, Israel
3
The School of Computer Science and Engineering and the Interdisciplinary Center for Neural Computation, The Hebrew University of Jerusalem, Jerusalem 9190401, Israel
*
Author to whom correspondence should be addressed.
Entropy 2020, 22(2), 208; https://doi.org/10.3390/e22020208
Received: 12 December 2019 / Revised: 9 February 2020 / Accepted: 10 February 2020 / Published: 12 February 2020
(This article belongs to the Special Issue Theory and Applications of Information Theoretic Machine Learning)
Canonical Correlation Analysis (CCA) is a linear representation learning method that seeks maximally correlated variables in multi-view data. Nonlinear CCA extends this notion to a broader family of transformations, which are more powerful in many real-world applications. Given the joint probability, the Alternating Conditional Expectation (ACE) algorithm provides an optimal solution to the nonlinear CCA problem. However, it suffers from limited performance and an increasing computational burden when only a finite number of samples is available. In this work, we introduce an information-theoretic compressed representation framework for the nonlinear CCA problem (CRCCA), which extends the classical ACE approach. Our suggested framework seeks compact representations of the data that allow a maximal level of correlation. This way, we control the trade-off between the flexibility and the complexity of the model. CRCCA provides theoretical bounds and optimality conditions, as we establish fundamental connections to rate-distortion theory, the information bottleneck and remote source coding. In addition, it allows a soft dimensionality reduction, as the compression level is determined by the mutual information between the original noisy data and the extracted signals. Finally, we introduce a simple implementation of the CRCCA framework, based on lattice quantization.
Keywords: canonical correlation analysis, alternating conditional expectation, remote source coding, dimensionality reduction, information bottleneck canonical correlation analysis, alternating conditional expectation, remote source coding, dimensionality reduction, information bottleneck
MDPI and ACS Style

Painsky, A.; Feder, M.; Tishby, N. Nonlinear Canonical Correlation Analysis:A Compressed Representation Approach. Entropy 2020, 22, 208.

Show more citation formats Show less citations formats
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop