Information Flows of Diverse Autoencoders

Abstract

Deep learning methods have had outstanding performances in various fields. A fundamental query is why they are so effective. Information theory provides a potential answer by interpreting the learning process as the information transmission and compression of data. The information flows can be visualized on the information plane of the mutual information among the input, hidden, and output layers. In this study, we examine how the information flows are shaped by the network parameters, such as depth, sparsity, weight constraints, and hidden representations. Here, we adopt autoencoders as models of deep learning, because (i) they have clear guidelines for their information flows, and (ii) they have various species, such as vanilla, sparse, tied, variational, and label autoencoders. We measured their information flows using Rényi’s matrix-based $\alpha$-order entropy functional. As learning progresses, they show a typical fitting phase where the amounts of input-to-hidden and hidden-to-output mutual information both increase. In the last stage of learning, however, some autoencoders show a simplifying phase, previously called the “compression phase”, where input-to-hidden mutual information diminishes. In particular, the sparsity regularization of hidden activities amplifies the simplifying phase. However, tied, variational, and label autoencoders do not have a simplifying phase. Nevertheless, all autoencoders have similar reconstruction errors for training and test data. Thus, the simplifying phase does not seem to be necessary for the generalization of learning.

1. Introduction

Since the development of information theory as a theory of communication by Shannon [1,2], it has played a crucial role in various domains of engineering and science, including physics [3], biology [4], and machine learning [5]. The information bottleneck (IB) theory interprets the learning process of neural networks as the transmission and compression of information [6]. Neural networks encode input X into internal representation Z; then, they decode Z to predict the desired output Y. The IB theory is a rate-distortion theory that compresses irrelevant information in X for predicting Y to the maximum extent. The objective function of this theory can be mathematically described as minimizing the mutual information $I(X;Z)$ between X and Z, given a required transmission $I_{req}$ of the mutual information $I(Z;Y)$ between Z and Y:

$$\begin{array}{c}\hfill \underset{p(Z|X)}{min}I(X;Z)-\beta \left[I(Z;Y)-I_{req}\right],\end{array}$$

(1)

where $\beta$ is a trade-off coefficient that balances the information compression and transmission. The numerical method for solving this optimization problem, so called Blahut–Arimoto algorithm [7,8], has been extensively studied using various problems [9,10]. Theoretical aspects of IB theory and its applications are well summarized in [11].

Here, it is important to note that machine learning models, including ours in this study, do not take Equation (1) as the loss function, although new deep variational IB models directly adopt it as the loss function [12]. The IB theory provides a nice interpretation of learning process, but it does not work for the optimization of neural networks in general. The mutual information provides a potent tool for visualizing the learning processes by displaying a trajectory on the two-dimensional plane of $I(X;Z)$ and $I(Z;Y)$, called the information plane (IP). Through IP analyses, Shwartz-Ziv and Tishby found that the training dynamics of neural networks demonstrate a transition between two distinct phases: fitting and compression [13,14]. Supervised learning experiences a short fitting phase in which the training error is significantly reduced. This first phase is characterized by increases in $I(X;Z)$ and $I(Z;Y)$. Then, in the learning process, a large amount of time is spent on finding the efficient internal representation Z of input X when the fitting phase secures a small training error. During this second phase of compression, $I(X;Z)$ decreases while $I(Z;Y)$ remains constant. To avoid unnecessary confusion with the usual data compression or dimensionality reduction, henceforth, we denote the second phase as “simplifying phase” instead of the original name of “compression phase”.

The simplifying phase has been argued to be associated with the generalization ability of machine learning by compressing irrelevant information of training data to prevent overfitting [14]. The non-trivial simplifying phase and its association with generalization have been further observed in other studies using different network models with different data; however, the universality of the simplifying phase remains debatable [15,16,17]. The debates can be partly attributed to the sensitivity toward the architecture of neural networks, activation functions, and estimation schemes of information measures.

In this study, we investigate how the information flows are shaped by the network designs, such as depth, sparsity, weight constraints, and hidden representations, by using autoencoders (AEs) as specific models of machine learning. AEs are neural networks that encode input X into internal representation Z and reproduce X by decoding Z. This representation learning can be interpreted as self supervised learning where a label is input as itself, such as $Y=X$. To examine the IP analyses of representation learning, we considered AEs because (i) they have a concrete guide ($Y=X$) for checking the validity of $I(X;Z)$ and $I(Z;Y)$ on the IP, (ii) they have various species to fully explore trajectories on the IP, and (iii) they are closely related to unsupervised learning.

The remainder of this paper is organized as follows. We introduce various types of AEs in Section 2 and explain our matrix-based kernel method for estimating mutual information in Section 3. Then, we examine the IP trajectories of information transmission and compression of the AEs in Section 4. Finally, we summarize and discuss our findings in Section 5.

2. Representation Learning in Autoencoders

2.1. Information Plane of Autoencoders

AEs are neural networks specialized for dimensional reduction and representation learning in an unsupervised manner. A deep AE consists of a symmetric structure with encoders and decoders as follows:

$$\begin{array}{c}\hfill X-E_1-\cdots -E_L-Z-D_1-\cdots -D_L-X^{\prime }.\end{array}$$

(2)

where $E_i$ and $D_i$ denote the i-th encoder and decoder layer, respectively, and Z is the bottleneck representation with the smallest dimension. The deep AE trains an identity function to reproduce input X from output $X^{\prime }$. During the training process, the AE extracts relevant features for reproducing X while compressing the high-dimensional input X into an internal representation Z on a low-dimensional bottleneck layer. The encoder and decoder layers form Markov chains that should satisfy the data processing inequality (DPI), analogously to supervised learning [18]:

$$\begin{array}{c}\hfill \mathrm{Forward}\mathrm{DPI}:\phantom{MMM}I(X;E_1)\ge \cdots \ge I(X;E_L)\ge I(X;Z),\end{array}$$

(3)

$$\begin{array}{c}\hfill \mathrm{Backward}\mathrm{DPI}:\phantom{MM}I(Z;X^{\prime })\le I(D_1;X^{\prime })\le \cdots \le I(D_L;X^{\prime }).\end{array}$$

(4)

The forward DPI represents information compression as input X is processed into the bottleneck layer, whereas the backward DPI represents information expansion as the compressed representation Z is transformed into output $X^{\prime }$. It is noteworthy that the usual AEs have physical dimensions, narrowing toward the bottleneck and expanding away from the bottleneck, which are consistent with the DPI.

The desired output of this AE is identical to the input ($X^{\prime }=X$). This identity constrains the input and output mutual information to be located on a straight line $I(X;T)=I(T;X)$ for arbitrary internal representations, $T\in \{E_1,\cdots ,E_L,Z,D_1,\cdots ,D_L\}$. Here, if the desired output X in $I(T;X)$ is replaced with the predicted output $X^{\prime }$ of the AE, the learning dynamics of the AE on the IP can be analyzed [18]. Then, the two sets of mutual information for representing information compression and transmission correspond to

$$\begin{array}{cc}\hfill I(X;T)& =H(T)-H(T|X)\end{array}$$

(5)

$$\begin{array}{cc}\hfill I(T;X^{\prime })& =H(T)-H(T|X^{\prime }),\end{array}$$

(6)

where $H(T)$ represents the Shannon entropy of T, and $H(T|X)$ and $H(T|X^{\prime })$ are the conditional entropies of T given X and $X^{\prime }$, respectively. The forward process of the AE realizes the deterministic mapping of T = $T(X)$ and $X^{\prime }$ = $X^{\prime }(T)$. Then, one-to-one correspondence of $X\to T$ implies no uncertainty for $H(T(X)|X)=0$, whereas the possibly many-to-one correspondence of $T\to X^{\prime }$ implies some uncertainty for $H(T|X^{\prime }(T))\ne 0$. Therefore, the inequality of $I(X;T)\ge I(T;X^{\prime })$ is evident because $H(T)\ge H(T)-H(T|X^{\prime })$, where the conditional entropy $H(T|X^{\prime })$ is non-negative. Based on this inequality, the learning trajectory of $I(X;T)$ and $I(T;X^{\prime })$ on the two-dimensional IP ($x,y$) can be expected to stay below the diagonal line $y=x$. Once the learning process of the AE is complete with $X^{\prime }=X$, the two sets of mutual information become equal to $I(X;T)=I(T;X^{\prime }=X)$, and the learning trajectory ends up on the diagonal line.

2.2. Various Types of Autoencoders

To investigate information flows of machine learning, we adopted AEs because their theoretical bounds of IP trajectories could guide our IP analysis. IP analysis has been used to visualize the information process in AEs [18,19]. Their IP trajectories satisfied the theoretical boundary of $I(X;T)\ge I(T;X^{\prime })$. Previous studies examined IP trajectories according to the size of the given bottleneck layers, but they did not investigate the associations between the simplifying phases and generalizations of AEs. Another important advantage of adopting AEs is their diverse variants that enable us to fully explore IP trajectories depending on the network designs, such as depth, sparsity, weight constraints, and hidden representations. In particular, because a certain AE model is directly linked to unsupervised learning, the model can be used to understand the information process of unsupervised learning. Now, we briefly introduce diverse species of AEs used in our experiments.

The simplest structure of AE, called shallow AE, consists of a single bottleneck layer between the input and output layers. In the shallow AE ($X-Z-X^{\prime }$), the forward propagation of input X is defined as

$$\begin{array}{cc}\hfill Z& =f_E(W_{E}X+b_E)\end{array}$$

(7)

$$\begin{array}{cc}\hfill X^{\prime }& =f_D(W_{D}Z+b_D),\end{array}$$

(8)

where W and b represent the weights and biases, respectively, and $f(s)$ is a corresponding activation function. Here, the subscripts E and D denote the encoder and decoder, respectively. The shallow AE is trained to minimize the reconstruction errors usually measured by the mean squared error (MSE) between output $X^{\prime }$ and desired output X. It has been analytically proven that a shallow AE with linear activation ($f(s)=s$) spans the same subspace as that spanned by principal component analysis (PCA) [20,21]. Deep AEs stack hidden layers in the encoder and decoder symmetrically; moreover, it is well known that deep AEs yield better compression than shallow AEs.

Right up till recently, a myriad of variants and techniques have been proposed to improve the performance of AEs via richer representations, such as sparse AE (SAE) [22], tied AE (TAE) [23], variational AE (VAE) [24], and label AE (LAE) [25,26].

• SAE was proposed to avoid overfitting by imposing sparsity in the latent space. The sparsity penalty is considered a regularization term of the Kullback–Leibler (KL) divergence between the activity of bottleneck layer Z and sparsity parameter $\rho$, a small value close to zero.
• TAE shares the weights for the encoder and decoder part ($W_E=W_D^T$), where superscript T depicts the transpose of a matrix. This model is widely used to reduce the number of model parameters while maintaining the training performance. Owing to its symmetrical structure, it can be interpreted as a deterministic version of restricted Boltzmann machines (RBMs), a representative generative model for unsupervised learning; consequently, the duality between TAE and RBM has been identified [27]. Compared to the vanilla AE, SAE and TAE have regularizations for the degrees of freedom for nodes and weights, respectively. Later, we visually validate how these constraints lead to a difference in the information flow of IP trajectories.
• The ultimate goal of AEs is to obtain richer expressions in the latent space. Therefore, an AE is not a mere replica model, but a generative model that designs a tangible latent representation to faithfully reproduce the input data as output. VAE is one of the most representative generative models with a similar network structure to AE; however, its mathematical formulation is fundamentally different. The detailed derivation of the learning algorithm of VAE is beyond the scope of this study, and thus it will be omitted [24]. In brief, the encoder network of VAE realizes an approximate posterior distribution $q_{\varphi }(Z|X)$ for variational inference, whereas the decoder network realizes a distribution $p_{\theta }(X|Z)$ for generation. The loss of VAE, known as the evidence lower bound (ELBO), is decomposed into a reconstruction error given by the binary cross entropy (BCE) between the desired output X and predicted output $X^{\prime }$, and the regularization of KL divergence between the approximate posterior distribution $q_{\varphi }(Z|X)$ and prior distribution $p(Z)$. As tangible Gaussian distributions are usually adopted as the approximate posterior and prior distributions of $q_{\varphi }(Z|X)$ and $p(Z)$, respectively, VAE has a special manifold of the latent variable Z.
• AEs do not use data labels. Instead, inputs work as self labels for supervised learning. Here, to design the latent space using label guides, we consider another AE, called label AE (LAE). LAE forces the input data to be mapped into the latent space with the corresponding label classifications. Then, the label-based internal representation is decoded to reproduce input data. Although the concept of regularization using labels has been proposed [25,26], LAE has not been considered as a generative model. Unlike vanilla AEs that use a sigmoid activation function, LAE uses a softmax activation function, $f(Z_i)=exp(Z_i)/{\sum }_{j}exp(Z_j)$, to impose the regularization of the internal representation Z to follow the true label Y as the cross entropy (CE) between Y and Z. Once LAE is trained, it can generate learned data or images using its decoder, starting from one-hot vector Z of labels with the addition of noise. Additional details of LAE are provided in Appendix B. Later, we compare the IP trajectories of VAE and LAE with those of vanilla AE in a deep structure to examine how the information flow varies depending on the latent space of generative models.

Table 1. Various species of autoencoder. Vanilla AE uses the mean squared error (MSE) loss. We used a sigmoid function as an activation function for the bottleneck layer, which helps in unifying the scales of different layers. Regularization of SAE is the KL-divergence between the hidden activity and sparsity parameter $\rho$. The only difference in TAE is that it shares the weight of encoder and decoder. The loss function of VAE, known as the evidence lower bound (ELBO), consists of the reconstruction error, binary cross entropy (BCE), and KL-divergence between the approximate posterior $q_{\varphi }(Z|X)$ and prior $p(Z)$; moreover, the stochastic node activities of the bottleneck layer are sampled from Gaussian distributions. In LAE, the classification error, the cross entropy (CE) between the softmax hidden activity Z and true label Y, is used as a regularization term.

Model	Main Loss	Constraint	Bottleneck Activation
AE	MSE$(X,X^{\prime })$	None	sigmoid
SAE	MSE$(X,X^{\prime })$	KL$(\rho ||Z)$	sigmoid
TAE	MSE$(X,X^{\prime })$	$W_E=W_D^T$	sigmoid
VAE	BCE$(X,X^{\prime })$	KL$(q_{\varphi }(Z|X)||p(Z))$	Gaussian sampling
LAE	MSE$(X,X^{\prime })$	CE$(Y,Z)$	softmax

summarizes the loss function, constraints, and activation function of the bottleneck layer for each aforementioned AE model.

3. Estimation of Mutual Information

After preparing various species of AE models to explore diverse learning paths on the IP, we need to estimate the mutual information for IP analyses:

$$\begin{array}{c}\hfill I(X;Z)=\sum _{x,z}p(x,z)log\frac{p(x,z)}{p(x)p(z)}.\end{array}$$

(9)

In reality, we have samples of data, ${\{x(t),z(t)\}}_{t=1}^N$, instead of their probabilities, $p(x),$$p(z)$, and $p(x,z)$. Using N samples of data, we may estimate the probabilities. Here, if X and Z are continuous variables, it is inevitable to first discretize them. Then, we can count the discretized samples for each bin and estimate the probabilities. The estimation of mutual information based on this binning method has some limitations. First, its accuracy depends on the resolution of discretization. Second, large samples are required to properly estimate the probability distributions. Suppose that X is an n-dimensional vector. Despite considering the most naive discretization with binarized activities, the total number of configurations for the binarized X is already $2^n$. Thus it becomes impracticable for N finite samples to cover the full range of configurations, e.g., $2^{20}\approx {10}^6$ configurations for $n=20$. Therefore, other schemes, such as kernel density estimation [28], k-nearest neighbors, and matrix-based kernel estimators [29,30], exist to estimate the entropy and mutual information. The description of each scheme and the corresponding IP results were presented in a pedagogical review [31]. Among these various methods, we adopted a matrix-based kernel estimator, which is mathematically well defined and computationally efficient for large networks. It estimates the Rényi’s $\alpha$-order entropy using the eigenspectrum of covariance matrix of X as follows:

$$\begin{array}{c}\hfill S_{\alpha }(A)=\frac{1}{1-\alpha }{log}_2\left[\mathrm{tr}(A^{\alpha })\right]=\frac{1}{1-\alpha }{log}_2\left[\sum _{i=1}^N{\lambda }_i{(A)}^{\alpha }\right],\end{array}$$

(10)

where A is an $N\times N$ normalized Gram matrix of random variable X with size N and ${\lambda }_i(A)$ is the i-th eigenvalue of A. Note that tr denotes the trace of a matrix. In the limit of $\alpha \to 1$, Equation (10) is reduced to an entropy-like measure that resembles the Shannon entropy of $H(X)$. If we assume that B is a normalized Gram matrix from another random variable Z, the joint entropy between X and Z is defined as

$$\begin{array}{c}\hfill S_{\alpha }(A,B)=S_{\alpha }\left(\frac{A\circ B}{\mathrm{tr}(A\circ B)}\right),\end{array}$$

(11)

where $A\circ B$ denotes the Hadamard product, i.e., the element-wise product of two matrices. From Equations (10) and (11), the mutual information can be defined as

$$\begin{array}{c}\hfill I_{\alpha }(X;Z)=S_{\alpha }(A)+S_{\alpha }(B)-S_{\alpha }(A,B),\end{array}$$

(12)

which is analogous to the standard mutual information in the new space called reproducing kernel Hilbert space (RKHS). Although $I_{\alpha }(X;Z)$ is mathematically different from $I(X;Z)$ in Equation (9), this quantity satisfies the mathematical requirements as Rényi’s entropy [29]. Furthermore, it has a great computational merit in that its computation is not affected much by the dimensiosn n of X, unlike the standard binning method for estimating the mutual information. In a simple setup where an exact computation of $I(X;Z)$ is possible, we confirmed that the matrix-based $I_{\alpha }(X;Z)$ gives an accurate estimation of $I(X;Z)$ (see Appendix A). Compared to the matrix-based estimator, the binning method gives less accurate results that are violently affected by the resolution of discretization and sample size. Using this estimator, Yu and Principe visualized the IP trajectories of AEs and suggested the optimal design of AEs based on IP patterns [18].

The kernel estimator contains a hyperparameter that defines a kernel function of distances between samples. As the estimator depends on the dimension and scale of variables for samples, the hyperparameter should be carefully determined [19]. Despite careful determination, the matrix-based kernel estimator seems unstable because it is sensitive to the training setup of neural networks. Moreover, once the information process of deep neural networks is quantified by this estimator, it sometimes violates the DPI, which is a necessary condition for interpreting layer stacks as Markov chains. We found that the raw activities of neural networks can result in inaccurate entropy estimations irrespective of the estimation schemes when they have different dimensions and scales depending on layers. Large activities tend to overestimate their entropies, whereas low activities tend to underestimate their entropies. In particular, the use of a linear activation function or rectified linear unit (ReLU) often results in the violation of DPI (see Figures 6 and 9 in [19]). To address this issue, we unified to use a bounded activation function, i.e., sigmoid function ($f(s)=1/(1+exp(-s)))$, except for VAE and LAE, whose bottleneck layers used Gaussian sampling and a softmax function, respectively; and this setup eliminated the DPI violation.

Saxe et al. argued that using double-sided saturating activation functions such as $f(s)=tanh(s)$ trivially induces the simplifying phase on the IP, and it is not related to the generalization of machine learning [15]. They showed that the mutual information, estimated by the binning method, first increases and then decreases as the weight parameters of neural networks get larger. The second decreasing phase of mutual information causes the simplifying phase. We performed the same task with various activation functions, including sigmoid and ReLU, but we estimated the mutual information using the aforementioned matrix-based kernel method. Then, we confirmed that the second decreasing phase did not occur by merely increasing the weight parameters, suggesting that the existence of the simplifying phase does not depend on the selection of activation functions in our matrix-based kernel method. Further details on this experiment are provided in Appendix A. For those who are interested in using IP analysis, we have provided the complete source code and documentation on GitHub [32].

4. Results

In this section, we examine the MI of various AE models using the method introduced in the previous section, and visualize it on IP. Our main concern is whether the phase transition in IP can be observed in representation learning. Furthermore, by comparing the IPs of different AEs, we investigate how the various techniques we adopted for efficient training of neural networks modified the information flow in latent space.

4.1. Information Flows of Autoencoders

In this study, we investigated the information process of representation learning for real image datasets (Figure 1a): MNIST [33], Fashion-MNIST [34], and EMNIST [35]. MNIST has 60,000 training and 10,000 testing images of $28\times 28$ pixels of 10 hand-written digits (0–9). Fashion-MNIST has the same data size as MNIST for 10 different fashion products, such as dresses and shirts. Finally, EMNIST is an extension of MNIST; it contains 10 digits and 26 uppercase (A–Z) and lowercase letters (a–z). In this paper, we focused on the results of MNIST because the results of Fashion-MNIST and EMNIST are basically the same (refer [32]).

For representation learning of MNIST, we first considered a shallow AE ($X-Z-X^{\prime }$) that included a single hidden or bottleneck layer (Figure 1b). The input, hidden, and output layers had $n_X=28\times 28=784,n_Z=50$, and $n_{X^{\prime }}=784$ nodes, respectively. We considered a fully-connected network between layers with the loss functions listed in Table 1. For the optimization of network weights, we used the stochastic gradient descent method with Adam optimization, given a batch size of 100 for a total of 50 epochs. With each learning iteration, the MSE$(X,X^{\prime })$ kept decreasing (Figure 1c). This implies that the output $X^{\prime }$ of AE successfully reproduced the input image X of training data. To measure the generalization ability of the AE, we examined the reproduction ability of the AE for test images that were not used in the learning process. We confirmed that the test error was as small as the training error. Given the faithful reproduction of input images, the indifferent error between training and test images defines successful generalization as usual. The IP trajectory of the AE during the learning process is presented in Figure 1d. As expected, the trajectory satisfies the inequality of $I_{\alpha }(X;Z)\ge I_{\alpha }(Z;X^{\prime })$, and ended up on their equality line because of $X^{\prime }\approx X$ at the end of training. As observed by Shwartz and Ziv [14], the IP trajectory showed two distinct phases of fitting and simplifying. In the initial fitting phase, the input mutual information $I_{\alpha }(X;Z)$ between X and Z increased. Then, during the second simplifying phase, $I_{\alpha }(X;Z)$ decreased. Note that this representation learning showed a simultaneous decrease in the output mutual information, whereas general supervised learning maintained the output mutual information as constant during the simplifying phase.

Next, we considered a deep AE ($X-E_1-E_2-Z-D_1-D_2-X^{\prime }$) with two additional encoder layers before the bottleneck layer and two decoder layers after the bottleneck layer (Figure 2a). The corresponding node numbers for the inner layers were $n_{E_1}=256$, $n_{E_2}=128,n_Z=50,n_{D_1}=128$, and $n_{D_2}=256$. The deep AE exhibited similar learning accuracy and generalization ability to the shallow AE (Figure 2b). During the learning process, we measured the mutual information using the matrix-based kernel estimator and confirmed that the learning process of the deep AE satisfied the DPI (Figure 2c). We observed the simplifying phase in the inner layers of $E_2,Z$, and $D_1$ (Figure 2d). However, the simplifying phase was not evident in the outer layers of $E_1$ and $D_2$ that had relatively large dimensions with high information capacity.

Subsequently, we explored whether the simplifying phase appeared even with a small amount of training data. Unless sufficient training data are provided, machine learning can easily overfit a small amount of training data and fail to generalize the test data. We conducted a learning experiment with the deep AE using 10% of the total training data. The training error kept decreasing, similarly to the training error given the full training data. However, the test error was significantly larger than the training error (Figure 2e). This demonstrates that the deep AE failed to generalize. After confirming the satisfaction of DPI (Figure 2f), we examined the IP trajectories. Unlike the results of full training data, we did not observe the simplifying phase from any layers (Figure 2g). Thus, this difference suggests that the simplifying phase seems to be associated with generalization by removing irrelevant details.

4.2. Sparse Activity and Constrained Weights

To examine the effect of regularization on the information flows, we considered different species of AEs that can modify the learning phases. SAE and TAE have additional regularization phases for node activities and weight parameters, respectively, in comparison to vanilla AE (Table 1). First, we examined SAE, which has the same structure as a shallow AE. SAE showed perfect learning and generalization (Figure 3a). It is of particular interest that the simplifying phase is markedly exaggerated in SAE (Figure 3b). The sparsity penalty of SAE turns off unnecessary activities of hidden nodes, which can accelerate the simplifying phase.

Second, we examined TAE, which also has the same structure as shallow AE and SAE, but has a weight constraint of $W_E=W_D^T$. Similarly to shallow AE and SAE, TAE showed perfect learning and generalization; however, its learning accuracy was slightly lower under the weight constraint (Figure 3c). However, TAE did not exhibit the simplifying phase (Figure 3d). This implies that the simplifying phase is not necessary for generalization. Given the weight constraint, TAE seems to have less potential capacity to remove irrelevant information than vanilla AE.

4.3. Constrained Latent Space

Now, we survey another species of AEs that more actively modify the latent space of the bottleneck layer, and further investigate the information flows in the learning process.

VAE is a generative model that maps input data X into a Gaussian distribution $q_{\varphi }(Z|X)$ for the latent variable Z. We considered a deep VAE that had the same structure ($X-E_1-E_2-Z-D_1-D_2-X^{\prime }$) as deep AE, and confirmed that the VAE can learn the training data of MNIST and generalize to reproduce the test data (Figure 4a). However, because VAE had a special constraint for the bottleneck layer, the information process from the input layer into the bottleneck layer did not satisfy the DPI (Figure 4b). The mutual information $I_{\alpha }(X;Z)$ between X and Z did not change during the training process. Indeed, the fixed value was close to the maximum entropy of X given its batch size of 100 samples, $I_{\alpha }(X;Z)\approx {log}_2(100)\approx 6.6$, which was independent of dimension $n_Z$ of Z (data not shown). It is noteworthy that the mutual information between X and Z did not change for the learning process, although the mapping $X\to Z$ kept reorganizing to distinguish the feature differences of X. This shows a limitation of the information measure $I_{\alpha }(X;Z)$, which failed to capture the content-dependent representation of Z. Besides the bottleneck layer, other layers still satisfied the DPI. Next, we displayed the IP trajectories of VAE for each layer (Figure 4c). We did not observe the simplifying phase in any layer in the VAE. Therefore, VAE can generalize without the simplifying phase, similarly to TAE.

LAE is another generative model that maps X into Z, where Z corresponds to label Y of X. Thus, unlike other AE models, LAE uses label information to shape its latent space. We used the same deep network structure as the deep AE and VAE for LAE. The deep LAE could also learn the training data of MNIST and generalize to reproduce the test data as well (Figure 4d). LAE satisfied the DPI (Figure 4e), and its IP trajectories also satisfied the ineqaulity of $I_{\alpha }(X;T)\ge I_{\alpha }(T;X^{\prime })$ (Figure 4f). It is interesting that LAE has orthogonal learning phases. LAE first increased the input mutual information $I_{\alpha }(X;T)$ for the encoding part. Once LAE arrived at a certain maximal $I_{\alpha }(X;T)$, the output mutual information $I_{\alpha }(T;X^{\prime })$ started to increase. This shows that LAE first extracts information from the input data relevant for the label classification, and then transfers information to output for reproducing input images. We found that the LAE did not exhibit the simplifying phase, even though it successfully generalized.

5. Discussion

We studied the information flows in the internal representations of AEs using a matrix-based kernel estimator. AEs are perfect models to investigate how the information flows are shaped during the learning process depending on the network designs, since they have diverse species with various depths, sparsities, weight constraints, and hidden representations. When we used sufficient training data, shallow and deep AEs demonstrated the simplifying phase, following the fitting phase, along with the generalization ability to reproduce test data, thereby confirming the original proposal by Shwartz-Ziv and Tishby [14]. However, when we used a small amount of training data to induce overfitting, the AEs did not exhibit a simplifying phase and generalization, suggesting that the simplifying phase is associated with generalization. When a sparsity constraint was imposed in the hidden activities of SAE, regularization amplified the simplifying phase and provided more efficient representations for generalization. However, the constraining weight parameters ($W_E=W_D^T$) of TAE showed perfect generalization in the absence of the simplifying phase. Furthermore, VAE and LAE, shaping the latent space with a variational distribution and label information, also achieved generalization without the simplifying phase. These counterexamples of TAE, VAE, and LAE clearly demonstrate that the simplifying phase is not necessary for the generalization of models.

It is noteworthy that the absence of the simplifying phase does not mean that compression does not occur in representation learning. When the encoder part has a narrowing architecture, information compression is inevitable, as demonstrated by the DPI. Then, the removal of irrelevant information from data may contribute to the generalization of models. After the completion of representation learning, AEs obtain a certain amount of mutual information $I_{\alpha }(X;Z)=I_{final}$ between the input data X and its internal representation Z. The paths that obtain $I_{final}$ seem different between AEs. In TAE, VAE, and LAE, $I_{\alpha }(X;Z)$ monotonically increases to $I_{final}$. However, in vanilla AE and SAE, $I_{\alpha }(X;Z)$ first increases beyond $I_{final}$, and then decreases back to $I_{final}$. The backward process is called the simplifying phase. As the loss function of representation learning never includes any instruction for the path of $I_{\alpha }(X;Z)$, it is not surprising that the existence of the simplifying phase is not universal. In summary, in the basic structure of AE, we found that the simplifying phase is related to the generalization of the model, and confirmed that learning dynamics of neural network can be interpreted with IB theory. However, for several variants of AE, no simplifying phase was observed, suggesting that all types of deep learning do not follow a universal learning dynamics.

Although observations and physical meanings of the phase transition in IP were contradictory in previous studies, it is still manifest that IP analysis is an excellent tool to monitor information transmission and compression inside the “black box” of neural networks. For IP analysis, accurate information estimation is a prerequisite. In general, it is difficult to calculate the entropies of high-dimensional variables, but we could solve this problem by estimating the physical quantities corresponding to the entropies in a kernel space. When we applied the estimator to the representation learning, we found that it is critical to use bounded activation functions. When we used ReLU as an activation function, the DPI was easily violated, although we observed the simplifying phase in this setting. Thus, it can be problematic to estimate the mutual information from unbounded variables with different scales for different layers. In this study, we provided concrete grounds to further explore the theoretical understanding of information processing in deep learning.