The Path from PCA to Autoencoders to Variational Autoencoders: Building Intuition for Deep Generative Modeling

Abstract

This tutorial provides a comprehensive and intuitive journey through the evolution of deep generative models, tracing a clear path from the foundations of Principal Component Analysis (PCA) to modern Variational Autoencoders (VAEs), showing how each method solves the limitations of the previous one. We begin with PCA, a linear tool for reducing data dimensions. Its inability to model non-linear patterns motivates the use of Autoencoders (AEs), which use neural networks to learn flexible, compressed representations. However, AEs lack a probabilistic framework, preventing them from generating new data. VAEs address this by treating the latent space as a probability distribution, enabling data generation. We compare the three methods through theoretical analysis, experiments, and step-by-step numerical examples that show exactly how each model compresses data—a detail often missing elsewhere. Unlike resources that treat these topics separately, we connect them into a single narrative, building intuition progressively from linear to probabilistic deep generative models.

1. Introduction

Dimensionality reduction (DR) stands as a cornerstone of machine learning (ML) and data science, solving the challenges of high-dimensional data. As data grows in scientific and industrial fields, the ′curse of dimensionality′ becomes a widespread obstacle, causing data sparsity, high computational costs, and lower model performance due to overfitting. Consequently, DR techniques are not merely a preprocessing step but a critical paradigm for data visualization, efficient computation, noise reduction, and finding the hidden low-dimensional structure that often underlies complex, high-dimensional observations [1].

There are two categories of DR techniques: supervised and unsupervised. Supervised methods use class labels to find patterns; unsupervised methods discover hidden structure without labels. This makes them useful for tasks like data visualization, compression, and noise removal. Principal Component Analysis (PCA) is a leading example, alongside other well-known techniques. Beyond this, DR methods can be categorized by whether the transformation is linear or non-linear. Linear methods like PCA are simpler; non-linear methods learn complex manifolds. Another criterion is the geometric property that they preserve. Some methods maintain global structure and large-scale distances, while others preserve local geometry and neighborhood relationships (e.g., Locally Linear Embedding (LLE) [2]). Finally, a modern and powerful taxonomy considers a model’s generative capability: descriptive vs. generative. Descriptive models (e.g. PCA and t-SNE) provide a compressed representation of data, but they do not learn its underlying probability distribution; thus, they cannot create new data. In contrast, generative models learn a model of the data distribution. This is achieved by learning a probabilistic mapping from the latent space back to the data space, enabling the generation of new data.

This tutorial focuses on a connected series of methods with a shared goal of learning efficient data representations: PCA, Autoencoders (AEs), and Variational Autoencoders (VAEs). As illustrated in Figure 1, these approaches differ fundamentally despite sharing dimensionality reduction as a goal: PCA uses linear transformation without learned parameters; AEs and VAEs use nonlinear transformations. An AE minimizes reconstruction error using learned neural network weights. VAEs, in contrast, introduce a probabilistic framework that maximizes the Evidence Lower Bound (ELBO) to enable generative modeling. This progression is driven by the need to address specific limitations. We begin with PCA [3], a linear method whose mathematical elegance and interpretability provide an ideal starting point. However, its linearity severely restricts its ability to model complex, real-world datasets. This limitation motivates the first jump to (non-linear) Autoencoders [4], which use neural networks to learn highly non-linear, compressed representations, offering a powerful and flexible framework for descriptive dimensionality reduction. However, standard AEs lack a probabilistic foundation, and their latent spaces are often poorly structured, making them unsuitable for generating new data. This key shortfall motivates the final conceptual leap to Variational Autoencoders [5]. By introducing a probabilistic interpretation of the latent space, VAEs transition from being tools for compression to being truly generative models. Thus, the journey from PCA to AEs to VAEs is one from linear to non-linear, from deterministic to probabilistic, and from compression to creation.

Beyond their direct application, the conceptual framework of VAEs has become a cornerstone of modern ML. The core ideas of VAEs—namely, a learned latent space, optimized inference, and the encoder–decoder structure—have been used in many subsequent developments. For example, diffusion models, as discussed in [6,7], are closely related to VAEs. Furthermore, the VAE objective provides a natural framework for learning disentangled representations [8,9], where independent latent factors control distinct, interpretable attributes of the data—a critical goal for explainable AI. This powerful architecture enables a wide range of practical applications, such as image generation [10], drug discovery [11], and anomaly detection [12]. The desire to achieve higher-quality generation has also led to the development of advanced variants such as Vector-Quantized VAEs (VQ-VAEs) [13], which play a crucial role in state-of-the-art systems for image and audio synthesis. The impact extends to semi-supervised learning [14], where the generative model improves learning from limited labeled data. Thus, a solid understanding of VAEs is essential to grasp a vast segment of modern AI, from generative models to representation learning. This extensive utility and theoretical elegance are core motivations for our tutorial.

The ML community has many educational resources that explain PCA, AEs, or VAEs. For instance, the mathematical foundations of PCA are explained in detail in foundational texts like [3,15]. Similarly, the concept of Autoencoders is explained thoroughly in deep learning textbooks [16], while the technical nuances of VAEs are the focus of dedicated, well-known tutorials such as [17] and the comprehensive survey in [18]. However, these resources, while excellent, often treat these models in isolation or with a focus on their standalone technical intricacies. Our tutorial distinguishes itself by connecting these topics into a single, coherent narrative that emphasizes their conceptual evolution. We start from the well-established and intuitive foundation of PCA and progressively build upon its limitations to motivate the introduction of non-linear AEs and, subsequently, the probabilistic framework of VAEs. This approach—tracing the path from a classic linear method to a modern deep generative model—builds intuition step by step. By minimizing the assumed prior knowledge and emphasizing the conceptual links over dense mathematical formalism, we aim to make the powerful ideas behind VAEs accessible and intuitive not only for ML specialists but also for researchers and students from any quantitative background.

The rest of this paper is organized as follows: Section 2 presents the theoretical background of DR methods and their categorization. We then trace the evolution from linear to non-linear probabilistic models across three core sections: Section 3 covers PCA, Section 4 introduces Autoencoders, and Section 5 presents VAEs. Section 3 details both covariance matrix and Singular Value Decomposition (SVD) approaches for dimensionality reduction and data reconstruction. This section includes step-by-step numerical examples demonstrating eigenvector selection and latent space construction. Section 4 explores the theoretical foundations of AEs, also supported by a clear numerical walkthrough of their architecture and training. Section 5 presents VAEs, providing a comprehensive, step-by-step explanation of their probabilistic framework. To bridge theory and practice, Section 6 offers two complementary experiments that empirically demonstrate the practical differences and progression from PCA to AE to VAE. Section 7 compares the key similarities and differences between the three methods from multiple perspectives. Finally, Section 8 provides concluding remarks and discusses future directions.

2. The Goal and Taxonomy of Dimensionality Reduction

A fundamental challenge in data analysis and ML is the curse of dimensionality, where data is often in a high-dimensional space, but its true structure is simpler and lies on a much lower-dimensional manifold. Dimensionality reduction (DR) addresses this by transforming data from the high-dimensional observation space into a meaningful lower-dimensional representation, known as the latent space.

Consider a dataset $\mathbf{X}=\{{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_N\}$, where each observation ${\mathbf{x}}_i\in {\mathbb{R}}^M$ is a vector with M-dimensional space (M features), and N is the total number of samples. The goal of a DR technique is to learn a mapping $f:{\mathbb{R}}^M\to {\mathbb{R}}^K$ that transforms each data point ${\mathbf{x}}_i$ into a latent representation ${\mathbf{z}}_i\in {\mathbb{R}}^K$ (where $K\ll M$), while preserving the essential information of the original data. The choice of K is a critical trade-off: a lower value achieves more compression but may discard important information, while a higher value preserves more structure but provides less compression. For instance, we project the data onto one dimension and compress each data point into a single scalar value. This mapping aims to find a low-dimensional representation that retains the maximum amount of meaningful information—such as statistical variance or shape—by eliminating redundancy. Because variables are often correlated, DR methods can eliminate this redundancy while keeping the most important patterns.

To understand different DR techniques, we can group them based on several key criteria. First, methods are either supervised or unsupervised. Supervised methods, such as Linear Discriminant Analysis (LDA) [19], use class labels to find the best low-dimensional subspace. In contrast, unsupervised methods, which are the focus of this tutorial, discover the hidden data structure without labels.

In addition, DR methods can be categorized by four critical features:

• Linearity: This distinguishes methods that project data onto a linear subspace (e.g., PCA [3]) from those that learn complex, non-linear manifolds (e.g., t-SNE [20]).
• Preservation Criterion: Methods that prioritize global structure and large-scale distances (e.g., Sammon mapping [21]) are contrasted with those that focus on preserving local neighborhoods (e.g., LLE [2]).
• Generative Capability: This criterion distinguishes descriptive models from generative models. Descriptive models, such as PCA and t-SNE, provide a compressed representation but cannot create new data. In contrast, generative models like VAEs explicitly learn the data distribution, enabling the generation of novel instances.
• Probabilistic vs. Deterministic: This distinction concerns the nature of the learned mapping. Deterministic methods (e.g., PCA) produce a fixed low-dimensional coordinate for each input. Probabilistic methods (e.g., VAEs) use probability distributions to model the latent space or the mapping. This quantifies uncertainty and provides a flexible, stochastic framework.

This taxonomy helps us see the conceptual evolution of DR methods, a progression defined by the nature of the mapping function. The journey begins with the simplicity of PCA, which uses linear projection. Since linear methods cannot capture complex relationships, we move to AEs, which employ non-linear functions—typically neural networks—to learn non-linear patterns. Finally, the need for a structured, generative latent space leads to VAEs, which introduce a probabilistic framework. In the following sections, we examine each method in detail, tracing this progression from linear compression to deep generative modeling.

3. Principal Component Analysis (PCA)

3.1. Definition of PCA

PCA is a standard linear dimensionality reduction technique. Its primary goal is to transform high-dimensional data into a lower-dimensional space while preserving as much information (variance) as possible [22].

PCA rotates the data onto new axes called principal components (PCs). These components define the PCA subspace ${\mathbf{Z}}_{\mathrm{PCA}}\subseteq {\mathbb{R}}^K$ with $K\le M$. The first principal component (PC₁) points in the direction of maximum variance. Each subsequent component is orthogonal to all previous ones and captures the next-largest variance (Figure 2). When we project the data onto PC₁ alone, the projected points exhibit higher variance than when projected onto PC₂. For instance, the points highlighted with a red circle—A and B—are farther apart when projected onto PC₁ than onto PC₂—because PC₁ preserves more variance.

3.2. PCA Space and Principal Components

The PCA space (${\mathbf{Z}}_{\mathrm{PCA}}$) is constructed using a set of K principal components. These components possess three key mathematical properties:

• Orthonormal: Each component vector has unit length ($\parallel {\mathbf{v}}_i\parallel =1$) and is perpendicular to all others. Mathematically,

  $$\begin{array}{c}\hfill {\mathbf{v}}_i^{\top }{\mathbf{v}}_j=\left\{\begin{array}{cc}1,\hfill & i=j\hfill \\ 0,\hfill & i\ne j\hfill \end{array}\right.\end{array}$$
  To understand this, if ${\mathbf{v}}_1=\left[\begin{array}{c}2\\ 0\end{array}\right]\mathrm{and}{\mathbf{v}}_2=\left[\begin{array}{c}0\\ 3\end{array}\right]$, then, ${\mathbf{v}}_1^{\top }{\mathbf{v}}_2=\left[\begin{array}{c}20\end{array}\right]\left[\begin{array}{c}0\\ 3\end{array}\right]=\left(2\right)\left(0\right)+\left(0\right)\left(3\right)=0$; hence, ${\mathbf{v}}_1$ and ${\mathbf{v}}_2$ are orthogonal (perpendicular), while

  $$\begin{array}{c}\hfill \parallel {\mathbf{v}}_1\parallel =\sqrt{2^2+0^2}=\sqrt{4}=2\ne 1,\parallel {\mathbf{v}}_2\parallel =\sqrt{0^2+3^2}=\sqrt{9}=3\ne 1\end{array}$$

  hence, they are orthogonal but not orthonormal, as they are not of unit length.
• Uncorrelated: The principal components are uncorrelated. This means that the covariance between different components is zero, $\mathrm{Cov}({\mathbf{v}}_i,{\mathbf{v}}_j)=0$ for $i\ne j$, where $\mathrm{Cov}({\mathbf{v}}_i,{\mathbf{v}}_j)$ represents the covariance between the $i^{th}$ and $j^{th}$ vectors.
• Ordered by Variance: The components are ordered so that ${\mathrm{PC}}_1$ captures the most variance, $P{C}_2$ captures the next most, and so on.

Due to the orthogonality of its components, PCA is an orthogonal transformation of the data. This can also be interpreted as a rotation of the original coordinate axes to align with the directions of the most spread-out data (i.e., maximal variance) [15,22].

3.3. PCA as an Optimization Problem

PCA can be understood in two equivalent ways: maximizing the variance of projected data, or minimizing the reconstruction error.

3.3.1. Variance Maximization Perspective

Consider a dataset of mean-centered data points $\{{\mathbf{d}}_1,{\mathbf{d}}_2,\dots ,{\mathbf{d}}_N\}$∈${\mathbb{R}}^M$ (where ${\mathbf{d}}_i={\mathbf{x}}_i-\mu$ and $\mu$ is the sample mean). We wish to find a unit vector $\mathbf{v}$ (where $\parallel \mathbf{v}\parallel =1$) such that the projection of the data onto $\mathbf{v}$ has the highest possible variance. For example, Figure 2 compares two vectors. As we can see, after assuming that both vectors are unit vectors, the projection onto ${\mathbf{v}}_1$ has higher variance than projecting the data onto ${\mathbf{v}}_2$.

The projection of a data point ${\mathbf{d}}_i$ onto $\mathbf{v}$ is the scalar ${\mathbf{v}}^{\top }{\mathbf{d}}_i$. Because the data is mean-centered data, the mean of the projected data is zero, and the variance of the projected data is therefore

$$\mathrm{Variance}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left({\mathbf{v}}^{\top }{\mathbf{d}}_i\right)}^2={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\mathbf{v}}^{\top }{\mathbf{d}}_i{\mathbf{d}}_i^{\top }\mathbf{v}={\mathbf{v}}^{\top }\left({\displaystyle \frac{1}{N}}\sum _{i=1}^N{\mathbf{d}}_i{\mathbf{d}}_i^{\top }\right)\mathbf{v}={\mathbf{v}}^{\top }\mathrm{\Sigma }\mathbf{v}$$

where $\mathrm{\Sigma }={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{\mathbf{d}}_i{\mathbf{d}}_i^{\top }$ is the covariance matrix (the scaling factor is limited to ${\displaystyle \frac{1}{N-1}}$, a factor that has no effect on the optimization). To find the best vector (first principal component), we solve the constrained optimization problem:

$$\underset{\mathbf{v}}{max}{\mathbf{v}}^{\top }\mathrm{\Sigma }\mathbf{v}\mathrm{subject}\mathrm{to}{\mathbf{v}}^{\top }\mathbf{v}=1.$$

Using the method of Lagrange multipliers, we define $\mathcal{L}(\mathbf{v},\lambda )={\mathbf{v}}^{\top }\mathrm{\Sigma }\mathbf{v}-\lambda ({\mathbf{v}}^{\top }\mathbf{v}-1)$, which leads to the condition that the optimal $\mathbf{v}$ must satisfy

$$\begin{array}{c}\hfill \mathrm{\Sigma }\mathbf{V}=\mathbf{V}\mathrm{\Lambda }\end{array}$$

(1)

where $\mathbf{V}$ is the matrix whose columns are the principal components (eigenvectors) of the covariance matrix $\mathrm{\Sigma }$, ordered by descending eigenvalue (from largest to smallest). That is, $\mathbf{V}=\{{\mathbf{v}}_1,{\mathbf{v}}_2,\dots ,{\mathbf{v}}_M\}$, where ${\mathbf{v}}_i$ is the eigenvector corresponding to the i-th largest eigenvalue. $\mathrm{\Lambda }$ is the diagonal matrix of corresponding eigenvalues; $\mathrm{\Lambda }=\mathrm{diag}({\lambda }_1,{\lambda }_2,\dots ,{\lambda }_M)$, where ${\lambda }_1\ge {\lambda }_2\ge \dots \ge {\lambda }_M$. Thus, Equation (1) represents the complete eigendecomposition of the covariance matrix, where the i-th column of $\mathbf{V}$ satisfies $\mathrm{\Sigma }{\mathbf{v}}_i={\lambda }_i{\mathbf{v}}_i$.

3.3.2. Reconstruction Error Perspective

Similarly, PCA can be derived by minimizing reconstruction error, which is the difference between the original data and the version that we get after compressing and then decompressing it. Consider encoding data into a latent space via ${\mathbf{z}}_i={\mathbf{Z}}_{\mathrm{PCA}}^{\top }{\mathbf{d}}_i$ and decoding back via ${\widehat{\mathbf{d}}}_i={\mathbf{Z}}_{\mathrm{PCA}}{\mathbf{z}}_i={\mathbf{Z}}_{\mathrm{PCA}}{\mathbf{Z}}_{\mathrm{PCA}}^{\top }{\mathbf{d}}_i$, where ${\mathbf{Z}}_{\mathrm{PCA}}$ is the matrix of principal components.

The objective is to minimize the squared reconstruction error across all data points:

$$\begin{array}{cc}\hfill J\left({\mathbf{Z}}_{\mathrm{PCA}}\right)& ={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\parallel {\mathbf{d}}_i-\widehat{{\mathbf{d}}_i}\parallel }^2\hfill \\ \hfill & ={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\parallel {\mathbf{d}}_i-{\mathbf{Z}}_{\mathrm{PCA}}{\mathbf{Z}}_{\mathrm{PCA}}^{\top }{\mathbf{d}}_i\parallel }^2\hfill \\ \hfill & ={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left[{\mathbf{d}}_i^{\top }{\mathbf{d}}_i-2{\mathbf{d}}_i^{\top }{\mathbf{Z}}_{\mathrm{PCA}}{\mathbf{Z}}_{\mathrm{PCA}}^{\top }{\mathbf{d}}_i+{\mathbf{d}}_i^{\top }{\mathbf{Z}}_{\mathrm{PCA}}\underset{=\mathbf{I}}{\underbrace{{\mathbf{Z}}_{\mathrm{PCA}}^{\top }{\mathbf{Z}}_{\mathrm{PCA}}}}{\mathbf{Z}}_{\mathrm{PCA}}^{\top }{\mathbf{d}}_i\right]\hfill \\ \hfill & ={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\mathbf{d}}_i^{\top }{\mathbf{d}}_i-{\displaystyle \frac{1}{N}}\sum _{i=1}^N{\mathbf{d}}_i^{\top }{\mathbf{Z}}_{\mathrm{PCA}}{\mathbf{Z}}_{\mathrm{PCA}}^{\top }{\mathbf{d}}_i=\mathrm{const}-\mathrm{trace}\left({\mathbf{Z}}_{\mathrm{PCA}}^{\top }\mathrm{\Sigma }{\mathbf{Z}}_{\mathrm{PCA}}\right)\hfill \end{array}$$

where ${\mathbf{Z}}_{\mathrm{PCA}}^{\top }{\mathbf{Z}}_{\mathrm{PCA}}=\mathbf{I}$ (orthogonality constraint) and the first term (${\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{\mathbf{d}}_i^{\top }{\mathbf{d}}_i$) is a constant independent of ${\mathbf{Z}}_{\mathrm{PCA}}$. Therefore, minimizing reconstruction error is equivalent to maximizing $\mathrm{trace}\left({\mathbf{Z}}_{\mathrm{PCA}}^{\top }\mathrm{\Sigma }{\mathbf{Z}}_{\mathrm{PCA}}\right)$, which is exactly the total variance of the projected data. The matrix ${\mathbf{Z}}_{\mathrm{PCA}}$ that maximizes this trace under orthonormality constraints consists of the eigenvectors of $\mathrm{\Sigma }$ corresponding to the K largest eigenvalues—the same principal components obtained from the variance maximization perspective. Thus, both viewpoints lead to the same solution.

There are two principal computational approaches for determining these principal components: one based on the eigendecomposition of the covariance matrix, and another utilizing the Singular Value Decomposition (SVD) of the data matrix.

3.4. Covariance Matrix Method

The covariance matrix method computes principal components in two steps: First, the covariance matrix of the data matrix $\mathbf{X}$ is computed. Second, an eigendecomposition is performed on this covariance matrix to find its eigenvalues and corresponding eigenvectors. The resulting eigenvectors form the principal components, ordered by the magnitude of their associated eigenvalues, which represent the amount of variance captured along each component. This computational pipeline is visualized in Figure 3.

3.4.1. Calculating Covariance Matrix ($\mathrm{\Sigma }$)

The covariance matrix helps us see how different variables (features) change together. For a single variable x, variance measures deviation from the mean: ${\sigma }^2=\mathbb{E}\left[{(x-\mu )}^2\right]=\mathbb{E}\left[x^2\right]-{\mu }^2$, where $\mu =\mathbb{E}\left[x\right]$ is the mean. For multiple variables, the covariance between variables $x_i$ and $x_j$ is

$$\begin{array}{c}\hfill {\mathrm{\Sigma }}_{ij}=\mathbb{E}\left[(x_i-{\mu }_i)(x_j-{\mu }_j)\right]=\mathbb{E}\left[x_i{x}_j\right]-{\mu }_i{\mu }_j\end{array}$$

As shown in Figure 3-step (A), the computational procedure begins by mean-centering the data matrix $\mathbf{X}$. For each variable, we compute its mean $\mu \in {\mathbb{R}}^M$, and then we subtract it from all samples:

$$\mathbf{D}=\{{\mathbf{d}}_1,{\mathbf{d}}_2,\dots ,{\mathbf{d}}_N\}=\{{\mathbf{x}}_1-\mu ,{\mathbf{x}}_2-\mu ,\dots ,{\mathbf{x}}_N-\mu \}$$

where $\mu ={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{\mathbf{x}}_i$.

Geometrically, the mean-centering data step represents a translation of the coordinate system to center the data around the origin, which is $\mu$ [15,23,24]. This translation ensures that (i) distances between points remain the same, (ii) angles between vectors remain the same, (iii) the shape of the data cloud remains the same, and (iv) the covariance structure remains the same. Meanwhile, the coordinate system’s origin moves to the data centroid, and the interpretation of coordinates also changes—they now represent deviations from the mean. This step is crucial in PCA, because PCA identifies directions of maximum variance, and variance is measured as the average squared distance from the mean: $\mathrm{Variance}=\mathbb{E}\left[{(x-\mu )}^2\right]$. After data centering, the mean $\mu$ becomes zero, which simplifies the variance calculation to $\mathbb{E}\left[x^2\right]$. This simplification occurs because when $\mu =0$, the expression ${(x-\mu )}^2$ reduces to $x^2$, making the computation more straightforward and computationally efficient. Similarly, the covariance matrix computation simplifies: $\mathrm{Cov}(\mathbf{X},\mathbf{Y})=\mathbb{E}\left[\mathbf{XY}\right]$ when the means are zero. Geometrically, all principal components pass through the origin.

The covariance matrix is then computed as the outer product of the mean-centered data (Figure 3-step (B)):

$$\begin{array}{c}\hfill \mathrm{\Sigma }={\displaystyle \frac{1}{N-1}}\sum _{i=1}^N{\mathbf{d}}_i{\mathbf{d}}_i^{\top }={\displaystyle \frac{1}{N-1}}\mathbf{D}{\mathbf{D}}^{\top }\end{array}$$

(2)

A valid covariance matrix is always (i) Symmetric: $\mathrm{\Sigma }={\mathrm{\Sigma }}^{\top }$ and (ii) Positive Semi-Definite: ${\mathbf{v}}^{\top }\mathrm{\Sigma }\mathbf{v}\ge 0$ for all $\mathbf{v}\ne \mathbf{0}$. This guarantees non-negative eigenvalues; geometrically, the data cloud forms an ellipse whose axes’ lengths are the square roots of the eigenvalues.

The structure of the covariance matrix reveals key statistical relationships:

$$\mathrm{\Sigma }=\left(\begin{array}{cccc}\mathrm{Var}\left({\mathbf{x}}_1\right)& \mathrm{Cov}({\mathbf{x}}_1,{\mathbf{x}}_2)& \dots & \mathrm{Cov}({\mathbf{x}}_1,{\mathbf{x}}_M)\\ \mathrm{Cov}({\mathbf{x}}_2,{\mathbf{x}}_1)& \mathrm{Var}\left({\mathbf{x}}_2\right)& \dots & \mathrm{Cov}({\mathbf{x}}_2,{\mathbf{x}}_M)\\ ⋮& ⋮& \ddots & ⋮\\ \mathrm{Cov}({\mathbf{x}}_M,{\mathbf{x}}_1)& \mathrm{Cov}({\mathbf{x}}_M,{\mathbf{x}}_2)& \dots & \mathrm{Var}\left({\mathbf{x}}_M\right)\end{array}\right)$$

The diagonal elements $\mathrm{Var}\left({\mathbf{x}}_i\right)$ represent the variances of individual variables ${\mathbf{x}}_i$ for $i=1,\dots ,M$, while the off-diagonal elements $\mathrm{Cov}({\mathbf{x}}_i,{\mathbf{x}}_j)$ capture the pairwise covariances between different variables. A positive covariance indicates a positive correlation between variables, negative values suggest inverse relationships, and zero values denote statistical uncorrelation or independence [15].

3.4.2. Eigenvalue–Eigenvector Decomposition for Principal Components

The fundamental mathematical operation underlying PCA involves solving the characteristic equation of the covariance matrix:

$$\mathrm{\Sigma }{\mathbf{v}}_i={\lambda }_i{\mathbf{v}}_i$$

(3)

where ${\lambda }_i$ denotes the i-th eigenvalue (a scalar quantity) and ${\mathbf{v}}_i$ represents its corresponding eigenvector (a non-zero vector).

In PCA, the eigenvectors (${\mathbf{v}}_i$) define orthogonal principal component directions of maximum variance, while their associated eigenvalues (${\lambda }_i$) quantify the amount of variance captured along each direction. We sort the eigenvectors from the largest eigenvalue to the smallest. This is because the largest eigenvalue points to the direction with the most information. To form our lower-dimensional PCA space, as in Figure 3-step (C), we choose the top K eigenvectors that correspond to the K largest eigenvalues ${\mathbf{Z}}_{\mathrm{PCA}}=[{\mathbf{v}}_1,{\mathbf{v}}_2,\dots ,{\mathbf{v}}_K]$ [25,26].

Special Case—Degenerate Eigenvalues: When multiple eigenvalues are equal (${\lambda }_i={\lambda }_j$), the corresponding principal components explain identical variance (the data is equally spread in both directions). Geometrically, the data cloud forms a perfectly symmetric shape in the subspace spanned by those eigenvectors. Hence, any orthogonal rotation of the associated eigenvectors is equally valid, which makes it harder to choose a “best” single component [25]. Such cases typically occur in datasets with spherical symmetry or when multiple features exhibit identical correlation structures.

3.5. Singular Value Decomposition (SVD)

SVD provides an alternative, more numerically stable way of calculating the principal components. SVD represents a fundamental matrix factorization technique in linear algebra, providing optimal low-rank approximations while preserving maximal variance (see Figure 3-step (D)). For PCA computation, we apply SVD to the mean-centered data matrix $\mathbf{D}\in {\mathbb{R}}^{M\times N}$$(\mathrm{where}\mathrm{each}\mathrm{column}{\mathbf{d}}_i\mathrm{represents}\mathrm{a}\mathrm{centered}\mathrm{data}\mathrm{point}{\mathbf{d}}_i^{\top }={({\mathbf{x}}_i-\mu )}^{\top })$ into three constituent matrices:

$$\mathbf{D}=\mathbf{L}\mathbf{S}{\mathbf{R}}^{\top }$$

(4)

where

• $\mathbf{L}\in {\mathbb{R}}^{M\times M}$ constitutes the left singular vectors; its columns form an orthonormal basis for the column space of $\mathbf{D}$ and are exactly the eigenvectors of $\mathbf{D}{\mathbf{D}}^{\top }$—the principal components.
• $\mathbf{S}\in {\mathbb{R}}^{M\times N}$ is a rectangular diagonal matrix containing singular values $s_1\ge s_2\ge \dots \ge s_r>0$, where $r=min(M,N)$. Singular values $s_i$ relate to eigenvalues via ${\lambda }_i={\displaystyle \frac{s_i^2}{N-1}}$.
• $\mathbf{R}\in {\mathbb{R}}^{N\times N}$ contains the right singular vectors (eigenvectors of ${\mathbf{D}}^{\top }\mathbf{D}$).

$\mathbf{L}$ and $\mathbf{R}$ are orthonormal bases, satisfying ${\mathbf{L}}^{\top }\mathbf{L}=\mathbf{I}$ and ${\mathbf{R}}^{\top }\mathbf{R}=\mathbf{I}$.

3.5.1. Equivalence of SVD and Covariance-Based PCA

This section establishes the fundamental relationship between SVD and eigendecomposition approaches for PCA. Both methods yield identical principal components and eigenvalues, providing complementary mathematical perspectives on the same underlying structure. As we mentioned in Section 3.4, the principal components of the mean-centered data matrix $\mathbf{D}$ are obtained through eigendecomposition of the covariance matrix $\mathrm{\Sigma }={\displaystyle \frac{1}{N-1}}{\mathbf{DD}}^{\top }$, which satisfies $\mathrm{\Sigma }\mathbf{V}=\mathbf{V}\mathrm{\Lambda }$, where $\mathbf{V}$ contains the eigenvectors and $\mathrm{\Lambda }$ the eigenvalues. Alternatively, applying SVD to the mean-centered data matrix yields $\mathbf{D}={\mathbf{LSR}}^{\top }$ (see Equation (4)).

The fundamental connection emerges when we express the covariance matrix in terms of SVD components:

$$\begin{array}{cc}\hfill \mathrm{\Sigma }& ={\displaystyle \frac{1}{N-1}}{\mathbf{DD}}^{\top }={\displaystyle \frac{1}{N-1}}\left({\mathbf{LSR}}^{\top }\right){\left({\mathbf{LSR}}^{\top }\right)}^{\top }\hfill \\ \hfill & ={\displaystyle \frac{1}{N-1}}\left({\mathbf{LSR}}^{\top }\right)\left({\mathbf{RSL}}^{\top }\right)={\displaystyle \frac{1}{N-1}}\mathbf{L}\mathbf{S}{\mathbf{R}}^{\top }\mathbf{R}\mathbf{S}{\mathbf{L}}^{\top }\hfill \end{array}$$

utilizing the orthonormality property ${\mathbf{R}}^{\top }\mathbf{R}=\mathbf{I}$:

$$\begin{array}{cc}\hfill \mathrm{\Sigma }& ={\displaystyle \frac{1}{N-1}}\mathbf{L}{\mathbf{S}}^2{\mathbf{L}}^{\top }\hfill \end{array}$$

This is the eigendecomposition $\mathrm{\Sigma }={\mathbf{LS}}^2{\mathbf{L}}^{\top }$, where $\mathrm{\Lambda }={\displaystyle \frac{1}{N-1}}{\mathbf{S}}^2$. Therefore:

• The principal components (eigenvectors of $\mathrm{\Sigma }$) are exactly the columns of $\mathbf{L}$.
• The eigenvalues relate to singular values via ${\lambda }_i={\displaystyle \frac{s_i^2}{N-1}}$. Equivalently, if we define the scaled data matrix $\tilde{\mathbf{D}}=\mathbf{D}/\sqrt{N-1}$, then the singular values ${\tilde{s}}_i$ of $\tilde{\mathbf{D}}$ satisfy ${\lambda }_i={\tilde{s}}_i^2$.

The proportion of variance explained by the i-th principal component is identical in both approaches:

$${\displaystyle \frac{{\lambda }_i}{{\sum }_{j=1}^r{\lambda }_j}}={\displaystyle \frac{{\sigma }_i^2}{{\sum }_{j=1}^r{\sigma }_j^2}}$$

where the total number of non-zero components is limited by $r=min(M,N)$, which is the smaller of the number of features or samples.

3.5.2. Computational and Numerical Considerations

A key advantage of the SVD approach lies in its superior numerical stability, particularly for challenging datasets. Unlike the covariance method, which explicitly forms and decomposes ${\mathbf{DD}}^{\top }$, SVD works directly on the centered data matrix ($\mathbf{D}$). This avoids numerical precision issues that commonly arise with the covariance method in three scenarios: when features have very different measurement scales, when variables are highly correlated, or when the number of features approaches or exceeds the number of observations. The SVD approach is generally preferred for high-dimensional data where $M\gg N$ due to its better numerical stability and efficiency. It is especially valuable for datasets with linear dependencies, where some features can be perfectly predicted from others. In such cases, the data matrix $\mathbf{X}\in {\mathbb{R}}^{M\times N}$ has reduced dimensionality (rank lower than $min(M,N)$), making the covariance matrix singular and causing traditional eigendecomposition to become unstable. SVD handles these situations gracefully without relying on matrix inversion, providing a more reliable path to the principal components [27,28,29].

3.6. Constructing the PCA Subspace

The PCA subspace (${\mathbf{Z}}_{\mathrm{PCA}}$) is constructed by selecting the K most significant principal components—those with the largest eigenvalues—as illustrated in Figure 3.

This selection criterion ensures maximum variance preservation from the original dataset, while excluding less informative components. The resulting low-dimensional subspace is defined as ${\mathbf{Z}}_{\mathrm{PCA}}=\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_K\}$, where each ${\mathbf{v}}_i$ represents a principal component direction.

Dimensionality reduction is achieved by projecting the mean-centered original data onto this optimized subspace:

$$\mathbf{Y}={\mathbf{Z}}_{\mathrm{PCA}}^{\top }\mathbf{D}$$

(5)

where, $\mathbf{Y}\in {\mathbb{R}}^{K\times N}$ denotes the transformed data in the reduced PCA space, as illustrated in Figure 4. This projection effectively discards $(M-K)$ dimensions from the original feature space while retaining the most statistically significant information. The total variance preserved is $\mathrm{Retained}\mathrm{Variance}={\sum }_{i=1}^K{\lambda }_i$, and the variance lost is $\mathrm{Lost}\mathrm{Variance}{\sum }_{i=K+1}^M{\lambda }_i$.

3.7. Data Reconstruction and Error Analysis

The original data can be reconstructed from its principal component representation through the inverse transformation:

$$\widehat{\mathbf{X}}={\mathbf{Z}}_{\mathrm{PCA}}\mathbf{Y}+\mu ={\mathbf{Z}}_{\mathrm{PCA}}{\mathbf{Z}}_{\mathrm{PCA}}^{\top }\mathbf{D}+\mu$$

(6)

where $\widehat{\mathbf{X}}$ is the reconstructed approximation of the original dataset $\mathbf{X}$ (each column: ${\mathbf{x}}_i={\mathbf{Z}}_{\mathrm{PCA}}{\mathbf{y}}_i+\mu$). The reconstruction error is typically measured as the sum of squared Euclidean distances between the original and reconstructed points:

$$\mathrm{Error}=\parallel \mathbf{X}-\widehat{\mathbf{X}}\parallel =\sum _{i=1}^N{\parallel {\mathbf{x}}_i-{\widehat{\mathbf{x}}}_i\parallel }^2.$$

(7)

Error decreases as we retain more principal components (i.e., larger K), because more variance is preserved. Crucially, the error exhibits an inverse relationship with the total variance captured by the selected principal components.

The quality of the PCA representation is often quantified by the explained variance ratio:

$$\mathrm{Explained}\mathrm{Variance}={\displaystyle \frac{\mathrm{Total}\mathrm{Variance}\mathrm{of}{\mathbf{Z}}_{\mathrm{PCA}}}{\mathrm{Total}\mathrm{Variance}}}={\displaystyle \frac{{\sum }_{i=1}^K{\lambda }_i}{{\sum }_{i=1}^M{\lambda }_i}}.$$

(8)

This ratio measures the proportion of the original data variance retained in the K-dimensional subspace [30]. A ratio of 1.0 means 100% variance is kept—this occurs when $K=M$ or when the discarded components have zero variance (perfectly correlated features).

3.8. PCA Algorithms

The first step in the PCA algorithm is to construct a data matrix $\mathbf{X}$, where each column represents a single sample and each row represents a specific feature. The detailed steps for calculating the principal components using both the covariance and SVD methods are summarized in Algorithm 1. Note that, in the SVD method, we use the scaled matrix $\tilde{\mathbf{D}}={\displaystyle \frac{\mathbf{D}}{N-1}}$ so that the squared singular values $s_i^2$ directly equal the eigenvalues ${\lambda }_i$.

The computational complexity of PCA is as follows:

• SVD-Based: $\mathcal{O}(min(M{N}^2,M^{2}N))$ for a data matrix $\mathbf{D}\in {\mathbb{R}}^{M\times N}$.
• Covariance-Based: $\mathcal{O}(M^{2}N+M^3)$ for the covariance computation plus the eigendecomposition.

Algorithm 1 Principal Component Analysis (PCA)

1: Input: Data matrix $\mathbf{X}=[{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_N]$, where ${\mathbf{x}}_i\in {\mathbb{R}}^M$ is the $i^{\mathrm{th}}$ sample.   2: Compute the sample mean vector: $\mu ={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{\mathbf{x}}_i$   3: Center the data matrix: $\mathbf{D}=[{\mathbf{d}}_1,{\mathbf{d}}_2,\dots ,{\mathbf{d}}_N]$, where ${\mathbf{d}}_i={\mathbf{x}}_i-\mu$   4: Compute Principal Components using one of the following methods:   5: if Using the Covariance Matrix then   6:    Compute the covariance matrix: $\mathrm{\Sigma }={\displaystyle \frac{1}{N-1}}\mathbf{D}{\mathbf{D}}^{\top }$   7:    Perform eigendecomposition: $\mathrm{\Sigma }\mathbf{V}=\mathbf{V}\mathrm{\Lambda }$   8:    Sort the eigenvectors in $\mathbf{V}$ by their corresponding eigenvalues in $\mathrm{\Lambda }$ (descending order).   9:    ${\mathbf{Z}}_{\mathrm{PCA}}=[{\mathbf{v}}_1,\dots ,{\mathbf{v}}_K]$ 10: else if Using the SVD then 11:    Construct the scaled matrix: $\tilde{\mathbf{D}}={\displaystyle \frac{1}{\sqrt{N-1}}}\mathbf{D}$ 12:    $[\mathbf{L},\mathbf{S},\mathbf{R}]=\mathrm{SVD}\left(\tilde{\mathbf{D}}\right)$ 13:    $\mathrm{Principal}\mathrm{Components}\mathrm{are}\mathrm{columns}\mathrm{of}\mathbf{L};\mathrm{Variances}\mathrm{are}{\lambda }_i=s_i^2$. 14:    ${\mathbf{Z}}_{\mathrm{PCA}}=[{\mathbf{l}}_1,\dots ,{\mathbf{l}}_K]$ 15: end if 16: Project the centered data onto the lower-dimensional space: $\mathbf{Y}={\mathbf{Z}}_{\mathrm{PCA}}^{\top }\mathbf{D}$. 17: Output: Projected data $\mathbf{Y}$ and projection matrix ${\mathbf{Z}}_{\mathrm{PCA}}$.

3.9. Numerical Examples

This section explains the steps of calculating a lower-dimensional space using the covariance matrix method through two numerical examples. The first example, adapted from [31], uses a two-feature dataset to visually illustrate the derivation of principal components. It details the computation of eigenvalues and eigenvectors and demonstrates the data projection and reconstruction processes. The second example employs a four-feature dataset to examine how PCA procedures are affected by increased dimensionality and to explore the influence of a constant variable with zero variance.

3.9.1. First Example: 2D Example

In this section, the lower-dimensional PCA space is constructed using the covariance matrix method. Consider a dataset with $N=8$ samples and $M=2$ features, as follows:

$$\mathbf{X}=\left[\begin{array}{cccccccc}1.00& 1.00& 2.00& 0.00& 5.00& 4.00& 5.00& 3.00\\ 3.00& 2.00& 3.00& 3.00& 4.00& 5.00& 5.00& 4.00\end{array}\right]$$

(9)

where each column vector ${\mathbf{x}}_i\in {\mathbb{R}}^2$ represents the $i^{\mathrm{th}}$ sample. The mean ($\mu =\left[\begin{array}{c}2.63\\ 3.63\end{array}\right]$) should be mean subtracted from data as follows, $\mathbf{D}=\mathbf{X}-\mu$, and the values of $\mathbf{D}$ will be as follows:

$$\mathbf{D}=\left[\begin{array}{cccccccc}-1.63& -1.63& -0.63& -2.63& 2.38& 1.38& 2.38& 0.38\\ -0.63& -1.63& -0.63& -0.63& 0.38& 1.38& 1.38& 0.38\end{array}\right]$$

(10)

The covariance matrix $\mathrm{\Sigma }$ is then computed according to Equation (2). Subsequently, its eigenvalues $\mathrm{\Lambda }$ and eigenvectors $\mathbf{V}$ are calculated. The resulting matrices are as follows:

$$\mathrm{\Sigma }={\displaystyle \frac{1}{N-1}}D{D}^{\top }=\left[\begin{array}{cc}3.71& 1.70\\ 1.70& 1.13\end{array}\right],\mathrm{\Lambda }=\left[\begin{array}{cc}0.28& 0.00\\ 0.00& 4.54\end{array}\right],\mathrm{and}\mathbf{V}=\left[\begin{array}{cc}0.45& -0.90\\ -0.90& -0.45\end{array}\right]$$

(11)

The results indicate that the second eigenvalue, ${\lambda }_2$, is larger than the first, ${\lambda }_1$. This eigenvalue accounts for approximately $4.54/(0.28+4.54)\approx 94.19\%$ of the total variance, while ${\lambda }_1$ accounts for only $0.28/(0.28+4.54)\approx 5.81\%$. This distribution reflects the greater significance of the second eigenvector compared to the first. Consequently, the second eigenvector (i.e., the second column of $\mathit{V}$) defines the direction of maximum variance and therefore constitutes the first principal component of the PCA space ($P{C}_1$). The variance explained by PC₁ is ${\displaystyle \frac{4.54}{0.28+4.54}}\approx 94.19\%$; PC₂ explains only $5.81\%$.

To reduce the dimensions from two to only one, the mean-centered data matrix $\mathbf{D}$ is then projected onto a lower-dimensional space. Here, we will select only one eigenvector as a lower-dimensional space, but which eigenvector we should select? Surely, we should select the one that has the maximum eigenvalue.

If we project the data onto ${\mathrm{PC}}_1$ (using ${\mathbf{v}}_2$) and ${\mathrm{PC}}_2$ (using ${\mathbf{v}}_1$), we get

$$\begin{array}{cc}\hfill {\mathbf{Y}}_{{\mathbf{v}}_1}& ={\mathbf{v}}_1^{\top }\mathbf{D},\mathrm{and}{\mathbf{Y}}_{{\mathbf{v}}_2}={\mathbf{v}}_2^{\top }\mathbf{D},\hfill \end{array}$$

where ${\mathbf{Y}}_{{\mathbf{v}}_1}$ and ${\mathbf{Y}}_{{\mathbf{v}}_2}$ represent the projection of $\mathbf{D}$ onto ${\mathbf{v}}_1$ (second principal component) and ${\mathbf{v}}_2$ (first principal component), respectively. The resulting projection vectors are as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\mathbf{Y}}_{{\mathbf{v}}_1}=\left[\begin{array}{cccccccc}-0.16& 0.73& 0.28& -0.61& 0.72& -0.62& -0.18& -0.17\end{array}\right]\\ \hfill {\mathbf{Y}}_{{\mathbf{v}}_2}=\left[\begin{array}{cccccccc}1.73& 2.18& 0.84& 2.63& -2.29& -1.84& -2.74& -0.50\end{array}\right]\end{array}\end{array}$$

(12)

When data are projected onto all eigenvectors and subsequently reconstructed, no information is lost, and the reconstruction error is zero. However, dimensionality reduction necessitates discarding information, as we did in this example by selecting only one eigenvector as a projection space (i.e., the projection space is one-dimensional) and discarding the other. If the discarded eigenvectors carry information, this removal will introduce a reconstruction error—a residual between the original and reconstructed data. The magnitude of this error, or residual, depends on the number of selected eigenvectors K and the variance they explain, as quantified by the corresponding eigenvalues. This error is defined as follows:

$${\widehat{\mathbf{X}}}_i={\mathbf{v}}_i{\mathbf{Y}}_{{\mathbf{v}}_i}+\mu .$$

The resulting reconstructions, ${\widehat{\mathbf{X}}}_1$ and ${\widehat{\mathbf{X}}}_2$, are presented below.

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\widehat{\mathbf{X}}}_1={\mathbf{v}}_1{\mathbf{Y}}_{{\mathbf{v}}_1}+\mu =\left[\begin{array}{cccccccc}2.55& 2.95& 2.75& 2.35& 2.95& 2.35& 2.55& 2.55\\ 3.77& 2.97& 3.37& 4.17& 2.98& 4.18& 3.78& 3.78\end{array}\right]\\ \hfill {\widehat{\mathbf{X}}}_2={\mathbf{v}}_2{\mathbf{Y}}_{{\mathbf{v}}_2}+\mu =\left[\begin{array}{cccccccc}1.07& 0.67& 1.88& 0.27& 4.68& 4.28& 5.08& 3.08\\ 2.85& 2.66& 3.25& 2.46& 4.65& 4.45& 4.84& 3.85\end{array}\right]\end{array}\end{array}$$

(13)

The reconstruction errors between the original data and the data reconstructed from the first and second eigenvectors are denoted by $E_{{\mathbf{v}}_1}$ and $E_{{\mathbf{v}}_2}$, respectively. The values of these errors are as follows:  

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill E_{{\mathbf{v}}_1}& =\mathbf{X}-{\widehat{\mathbf{X}}}_1=\left[\begin{array}{cccccccc}-1.55& -1.95& -0.75& -2.35& 2.05& 1.65& 2.45& 0.45\\ -0.77& -0.97& -0.37& -1.17& 1.02& 0.82& 1.22& 0.22\end{array}\right]\hfill \\ \hfill E_{{\mathbf{v}}_2}& =\mathbf{X}-{\widehat{\mathbf{X}}}_2=\left[\begin{array}{cccccccc}-0.07& 0.33& 0.12& -0.27& 0.32& -0.28& -0.08& -0.08\\ 0.15& -0.66& -0.25& 0.54& -0.65& 0.55& 0.16& 0.15\end{array}\right]\hfill \end{array}\end{array}$$

(14)

When we reconstruct the data using only ${\mathrm{PC}}_1$, the error for the first sample is ${([1.00;3.00]-[1.07;2.85])}^2={(-0.07)}^2+{\left(0.15\right)}^2\approx 0.03$. However, if we had mistakenly used ${\mathrm{PC}}_2$ (the component with less variance), the error would be 3.00. The total error for the dataset using ${\mathrm{PC}}_1$ is 1.98, while using ${\mathrm{PC}}_2$ it is 31.77. This confirms that ${\mathrm{PC}}_1$ preserves the most information.

Figure 5 provides a visual comparison of the two principal components. The original data samples, represented as red stars in Figure 5 (left), are points in a two-dimensional space (${\mathbf{x}}_i\in {\mathbb{R}}^2$). The two eigenvectors (${\mathbf{v}}_1$ and ${\mathbf{v}}_2$) derived from the data in this example are depicted as lines. The solid line represents the second eigenvector (${\mathbf{v}}_2$), which corresponds to the first principal component ($P{C}_1$), while the dotted line represents the first eigenvector (${\mathbf{v}}_1$), which corresponds to the second principal component ($P{C}_2$). The projection of the original data onto these two principal components is shown in Figure 5 (right). This figure visualizes the individual sample errors $E_{{\mathbf{v}}_1}$ and $E_{{\mathbf{v}}_2}$ with green and blue lines, respectively. As shown, the error associated with the second eigenvector ($E_{{\mathbf{v}}_2}$) is consistently and substantially lower than that of the first ($E_{{\mathbf{v}}_1}$).

3.9.2. Second Example: 4D Example

In this example, each sample is represented by four features ($M=4$), with the second feature being constant across all observations (the highlighted row):

$$\mathbf{X}=\left[\begin{array}{c}1.001.002.000.007.006.007.008.00\\ 2.002.002.002.002.002.002.002.00\\ 5.006.005.009.001.002.001.004.00\\ 3.002.003.003.004.005.005.004.00\end{array}\right]$$

(15)

The covariance matrix $\mathrm{\Sigma }$ is calculated as follows:

$$\mathrm{\Sigma }=\left[\begin{array}{cccc}10.86& 0.00& -7.57& 2.86\\ 0.00& 0.00& 0.00& 0.00\\ -7.57& 0.00& 7.55& -2.23\\ 2.86& 0.00& -2.23& 1.13\end{array}\right]$$

(16)

From the above covariance matrix, the zero values in the second row and column reflect the zero variance of the second feature due to its constant value.

The eigenvalues $\mathrm{\Lambda }$ and eigenvectors $\mathit{V}$ of the covariance matrix are as follows:

$$\mathrm{\Lambda }=\left[\begin{array}{cccc}17.75& 0.00& 0.00& 0.00\\ 0.00& 1.46& 0.00& 0.00\\ 0.00& 0.00& 0.33& 0.00\\ 0.00& 0.00& 0.00& 0.00\end{array}\right],\mathit{V}=\left[\begin{array}{cccc}0.76& 0.62& -0.20& 0.00\\ 0.00& 0.00& 0.00& 1.00\\ -0.61& 0.79& 0.10& 0.00\\ 0.21& 0.05& 0.98& 0.00\end{array}\right]$$

(17)

Several key observations emerge from these results. First, the first eigenvector accounts for ${\displaystyle \frac{17.75}{17.75+1.46+0.33+0}}\approx 90.84\%$ of the total variance, making it the first principal component. Second, the first three eigenvectors collectively represent $100\%$ of the total variance. This makes the fourth eigenvector, which is associated with an eigenvalue of zero, mathematically redundant, as it corresponds to a constant dimension across all samples. Third, as in Equation (18), projection onto the fourth eigenvector captures no meaningful variation and results in a high reconstruction error of ≈136.75. In contrast, projection onto the first three eigenvectors yields errors of ≈$12.53$, $126.54$, and $134.43$, respectively, directly reflecting their decreasing contribution to the explained variance.

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathbf{Y}}_{{\mathbf{v}}_1}& =\left[\begin{array}{cccccccc}-2.95& -3.78& -2.19& -6.16& 4.28& 3.12& 4.49& 3.20\end{array}\right]\hfill \\ \hfill {\mathbf{Y}}_{{\mathbf{v}}_2}& =\left[\begin{array}{cccccccc}-1.20& -0.46& -0.58& 1.32& -0.58& -0.39& -0.54& 2.39\end{array}\right]\hfill \\ \hfill {\mathbf{Y}}_{{\mathbf{v}}_3}& =\left[\begin{array}{cccccccc}0.06& -0.82& -0.14& 0.64& -0.52& 0.75& 0.45& -0.43\end{array}\right]\hfill \\ \hfill {\mathbf{Y}}_{{\mathbf{v}}_4}& =\left[\begin{array}{cccccccc}0.00& 0.00& 0.00& 0.00& 0.00& 0.00& 0.00& 0.00\end{array}\right]\hfill \end{array}\end{array}$$

(18)

4. Autoencoder (AE)

PCA is mathematically elegant, but its linearity limits its ability to model complex data distributions. This leads us to a more expressive framework: the Autoencoder (AE). Autoencoders were first proposed by Rumelhart, Hinton, and Williams in 1986, with the goal of learning to reconstruct input data with as little error as possible [5,32].

The Autoencoder is a neural network architecture designed to learn efficient non-linear data encodings in an unsupervised manner. Its goal is not just to reduce dimensions but to find the hidden, “deep” structure of the data.

The Autoencoder architecture consists of three fundamental components:

1.

Encoder: The encoder $f_{\varphi }:{\mathbb{R}}^M\to {\mathbb{R}}^K$ is a function with learnable parameters $\varphi =\{{\mathbf{W}}_E,{\mathbf{b}}_E\}$ (weights and biases). It maps input data (${\mathbf{x}}_i$) to a lower-dimensional latent code ${\mathbf{h}}_i=f_{\varphi }\left({\mathbf{x}}_i\right)$, where ${\mathbf{h}}_i\in {\mathbb{R}}^K$ represents the latent feature representation. These “latent features” are a compressed summary of the input. For example, latent features for hand-written digits could encode information about the number of lines required to write each number, the angles of these lines, and how they connect. For a single hidden layer architecture,

$${\mathbf{Z}}_{\mathrm{AE}}=f_{\varphi }\left(\mathbf{X}\right)=\sigma ({\mathbf{W}}_E\mathbf{X}+{\mathbf{b}}_E),$$

where $\sigma$ denotes a non-linear activation function like ReLU, tanh (for more details about activation functions, see Appendix A). These non-linearities allow the network to learn much more complex patterns than PCA.

In deep architectures, the encoder has multiple layers (see Figure 6, where the encoder consists of two layers) with progressively decreasing dimensions: the first layer has $n_1=M$ neurons (matching the input size), and the number of neurons in each layer drops, with the middle (or bottleneck) layer having the smallest number of neurons.

$${\mathbf{h}}^{\left(1\right)}=\sigma ({\mathbf{W}}_E^{\left(1\right)}\mathbf{x}+{\mathbf{b}}_E^{\left(1\right)}),{\mathbf{h}}^{\left(K\right)}=\sigma ({\mathbf{W}}_E^{\left(K\right)}{\mathbf{h}}^{(K-1)}+{\mathbf{b}}_E^{\left(K\right)})$$

with ${\mathbf{Z}}_{\mathrm{AE}}={\mathbf{h}}^{\left(L_E\right)}$, where $L_E$ is the number of encoder layers [32].

2.

Latent Space: The latent code ${\mathbf{Z}}_{\mathrm{AE}}\in {\mathbb{R}}^K$ represents the output of the encoder and contains a compressed representation of the input data, where $K\ll M$. This bottleneck is the critical architectural element that prevents the network from learning a trivial identity mapping, forcing it to prioritize the most important features. Unlike the probabilistic latent space in VAEs, the standard Autoencoder’s latent space is deterministic, meaning that for each input, the encoder always produces exactly the same latent representation—there is no randomness or probability distribution involved in the encoding process [33].

3.

Decoder: The decoder $g_{\theta }:{\mathbb{R}}^K\to {\mathbb{R}}^M$ reconstructs data from the latent code through the transformation ${\widehat{\mathbf{x}}}_i=g_{\theta }\left({\mathbf{h}}_i\right)=g_{\theta }\left(f_{\varphi }\left({\mathbf{x}}_i\right)\right)$. The decoder $g_{\theta }$, parameterized by $\theta =\{{\mathbf{W}}_D,{\mathbf{b}}_D\}$, performs the inverse mapping from the latent space back to the original data space:

$$\widehat{\mathbf{x}}=g_{\theta }\left({\mathbf{Z}}_{\mathrm{AE}}\right)=\sigma ({\mathbf{W}}_D{\mathbf{Z}}_{\mathrm{AE}}+{\mathbf{b}}_D).$$

For deep architectures, the decoder mirrors the encoder structure with progressively expanding layers:

$${\mathbf{h}}^{(L_E+1)}=\sigma ({\mathbf{W}}_D^{\left(1\right)}\mathbf{z}+{\mathbf{b}}_D^{\left(1\right)}),\widehat{\mathbf{x}}=\sigma ({\mathbf{W}}_D^{\left(L_D\right)}{\mathbf{h}}^{(L_E+L_D-1)}+{\mathbf{b}}_D^{\left(L_D\right)})$$

where $L_D$ denotes the number of decoder layers.

The learning objective is to optimize parameters $\varphi$ and $\theta$ by minimizing a reconstruction loss:

$$\underset{\varphi ,\theta }{min}\mathcal{L}(\mathbf{x},\widehat{\mathbf{x}})=\underset{\varphi ,\theta }{min}\mathcal{L}(\mathbf{x},g_{\theta }\left(f_{\varphi }\left(\mathbf{x}\right)\right))$$

where $\widehat{\mathbf{x}}$ is the reconstructed output. The network learns a function that, for a given input $\mathbf{x}$, produces a reconstruction $\widehat{\mathbf{x}}$ from a compressed latent code. Critically, this learning is constrained by a bottleneck: the low-dimensional latent space (with dimension $K\ll M$). By minimizing this loss, the network learns to keep only the features that are absolutely necessary to rebuild the input.

4.1. Parameters and Training

The complete parameter set of an Autoencoder is $\Theta =\{\varphi ,\theta \}$. Key architectural hyperparameters include the following:

• Network Architecture: Autoencoders typically employ symmetric architectures where the decoder mirrors the encoder. This means that the decoder network has the same number of layers as the encoder, with corresponding layers having matching sizes in reverse order, as shown in Figure 6. For example, with 784-pixel MNIST images, if the encoder goes from Input(784) → (256) → (64), the decoder usually goes from (64) → FC(256) → Output(784) [33].
• Latent Space Dimensionality: The size K of the latent code ${\mathbf{Z}}_{\mathrm{AE}}$ represents a critical trade-off. If K is too small, excessive information is lost, while if K is too large, the network can avoid meaningful compression.
• Loss Function: The selection of an appropriate loss function should be guided by the data characteristics, desired reconstruction properties, and the specific application requirements, as these fundamentally shape what the autoencoder learns to preserve and discard during compression. Different loss functions make different assumptions about the data distribution and error sensitivity, leading to substantially different learned representations and reconstruction behaviors. Here, we focus on this study of the data and how this affects the choice of reconstruction loss $\mathcal{L}(\mathbf{x},\widehat{\mathbf{x}})$. The most common choice for real-valued data is the Mean Squared Error (MSE) loss, also known as $L_2$ loss. MSE measures the average squared difference between the original input $\mathbf{x}$ and its reconstruction $\widehat{\mathbf{x}}$:

  $$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{MSE}}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\parallel {\mathbf{x}}_i-{\widehat{\mathbf{x}}}_i\parallel }^2\end{array}$$

Parameters $\Theta$ are optimized via gradient-based methods (e.g., Adam, SGD) through backpropagation to minimize the reconstruction loss over the training dataset.

4.2. The Generative Limitation

Although Autoencoders excel at compression, they are not effective at generating new data. This is because the latent space is deterministic and unstructured, which often results in “gaps”. If you select a random point in the latent space that the network did not encounter during training, the decoder will probably generate nonsense rather than a realistic image. This lack of a smooth, probabilistic structure is why VAEs are necessary.

4.3. Practical Considerations for Autoencoders

In Autoencoder design, the choice of activation function depends on the specific layer and data characteristics:

• Encoder Hidden Layers: The function of the encoder is to perform compression and feature extraction, mapping high-dimensional inputs to low-dimensional latent codes. This process relies on activations that promote sparsity, computational efficiency, and gradient flow for effective feature learning. As a result, the Rectified Linear Unit (ReLU) and its variants have become the standard choice for the hidden layers of the encoder. This preference stems from ReLU’s computational efficiency, which involves only a simple thresholding operation, and its effectiveness in mitigating the vanishing gradient problem in deep networks (more details on vanishing gradients and activation functions can be found in Appendix A). However, ReLU neurons can occasionally “die,” outputting zero for all inputs and resulting in inactive neurons. Alternatives such as Leaky ReLU and Exponential Linear Unit (ELU) can help address this issue by permitting small negative values.
• Decoder Hidden Layers: The function of the decoder is to perform reconstruction and generation, mapping the low-dimensional latent code back to high-dimensional output. This task requires activations that facilitate smooth interpolation, stable gradient flow for reconstruction quality, and controlled output ranges—especially near the final layer. In this context, the hyperbolic tangent function (tanh) often outperforms ReLU in deeper decoders due to three key properties: (1) bounded outputs prevent extreme activations that can destabilize training, (2) smooth differentiability enhances gradient flow, and (3) zero-centered symmetry aids in reconstructing zero-centered data. Both ReLU and hyperbolic tangent (tanh) activations are commonly used in the hidden layers of the decoder. The choice of activation function depends on the nature of the data and the output requirements. ReLU is often preferred for its simplicity and non-saturating behavior, which can be beneficial for training deep decoders. However, for data that is zero-centered (e.g., normalized to $[-1,1]$), tanh may be more suitable due to its zero-centered and bounded output (more details about the zero-centered problem can be found in Appendix A), potentially leading to more stable training.
  In practice, the selection of activation functions in the decoder is frequently influenced by empirical performance on the validation set.
• Output Layer: The activation function of the decoder’s output layer is chosen to match the statistical properties of the input data being reconstructed. This ensures that the reconstruction $\widehat{\mathbf{x}}$ resides in the same domain as the original input $\mathbf{x}$.

  -

  Sigmoid Activation: Used for data where each element represents a probability or intensity normalized to $[0,1]$. For example, binary-valued data (e.g., black-and-white images) and grayscale images normalized to $[0,1]$ (e.g., MNIST, Fashion-MNIST).

  -

  Softmax Activation: Used for discrete categorical data or one-hot encoded features. This is particularly important when modeling multinomial distributions (e.g., pixel intensities discretized into 256 bins), reconstructing text sequences or one-hot encoded categorical variables, or each output dimension must sum to 1 across categories.

  -

  Tanh Activation: Used for data normalized to the range $[-1,1]$, which is common when working with preprocessed image datasets where zero-centered data is beneficial.

  -

  Linear Activation (no activation): Used for unconstrained real-valued data where outputs can theoretically range over $(-\infty ,+\infty )$. Examples include regression tasks, scientific measurements without bounded ranges, and certain types of continuous sensor data.

• Latent Space: The activation function for the encoder’s output layer defines the latent space’s properties. A linear activation is most common, producing an unconstrained, interpretable real-valued vector analogous to PCA components. For specific applications requiring bounded latent codes (e.g., interpretable normalized features), a tanh or sigmoid activation can be used to restrict the latent values to (−1, 1) or (0, 1), respectively.

4.4. Numerical Example

To illustrate this, we will follow one sample through a simple Autoencoder. We will train this network using the same first sample from the PCA example, ${\mathbf{x}}_1={\left[\begin{array}{c}1.003.00\end{array}\right]}^{\top }$, to facilitate comparison between the two dimensionality reduction methods. The goal is to demonstrate, step by step, how Autoencoders compress input data into a lower-dimensional latent representation and then reconstruct it, all while minimizing reconstruction error through gradient-based optimization. This process highlights how neural networks can learn similar low-dimensional representations through iterative optimization.

Consider a symmetric Autoencoder designed to learn a one-dimensional representation from two-dimensional input data. This architecture mirrors the dimensionality reduction objective of PCA but achieves it through a different, parameterized approach. The network consists of five layers (see Figure 7) arranged as follows:

• Input Layer: Two neurons (accepting 2D input vectors).
• Encoder Hidden Layer: One neuron with hyperbolic tangent (tanh) activation.
• Bottleneck Layer: One neuron with linear activation (producing the 1D latent code).
• Decoder Hidden Layer: One neuron with tanh activation.
• Output Layer: Two neurons with linear activation (reconstructing the original 2D input).

For clarity in tracking computations, we initialize all parameters with simple values:

• Layer 1 (Encoder): ${\mathbf{W}}^{\left(1\right)}=\left[\begin{array}{cc}0.3& 0.4\end{array}\right],{\mathbf{b}}^{\left(1\right)}=0.1$
• Layer 2 (Bottleneck): ${\mathbf{W}}^{\left(2\right)}=\left[\begin{array}{c}0.5\end{array}\right],{\mathbf{b}}^{\left(2\right)}=0.2$
• Layer 3 (Decoder): ${\mathbf{W}}^{\left(3\right)}=\left[\begin{array}{c}0.6\end{array}\right],{\mathbf{b}}^{\left(3\right)}=0.3$
• Layer 4 (Output): ${\mathbf{W}}^{\left(4\right)}=\left[\begin{array}{c}0.7\\ 0.8\end{array}\right],{\mathbf{b}}^{\left(4\right)}=\left[\begin{array}{c}0.4\\ 0.5\end{array}\right]$

In this example, we used the tanh activation function to introduce non-linearity, while linear activation functions were employed in the bottleneck and output layers.

4.4.1. Forward Propagation

The forward propagation process demonstrates how the Autoencoder transforms input data through its layers, producing both a compressed representation and a reconstruction. This process consists of four sequential computations, each corresponding to one layer of the network.

Step 1: Encoder Hidden Layer Computation: The first step compresses the 2D input into the hidden representation using a linear transformation followed by the tanh activation:

$$\begin{array}{c}\hfill {\mathbf{z}}^{\left(1\right)}={\mathbf{W}}^{\left(1\right)}{\mathbf{x}}_1+{\mathbf{b}}^{\left(1\right)}=[0.30.4]\left[\genfrac{}{}{0ex}{}{1.00}{3.00}\right]+0.1=1.6\Rightarrow {\mathbf{a}}^{\left(1\right)}=\tanh \left({\mathbf{z}}^{\left(1\right)}\right)\approx 0.9217\end{array}$$

Here, ${\mathbf{z}}^{\left(1\right)}$ represents the pre-activation value, and ${\mathbf{a}}^{\left(1\right)}$ is the activated output. The tanh function squashes this value to the range $(-1,1)$, introducing non-linearity that allows the network to learn complex mappings.

Step 2: Bottleneck Layer Computation: This step produces the one-dimensional latent representation, which serves as the compressed code for the input:

$$\begin{array}{c}\hfill {\mathbf{a}}^{\left(2\right)}={\mathbf{z}}^{\left(2\right)}={\mathbf{W}}^{\left(2\right)}{\mathbf{a}}^{\left(1\right)}+{\mathbf{b}}^{\left(2\right)}=0.5\times 0.9217+0.2=0.6609\end{array}$$

where the bottleneck layer uses linear activation, meaning ${\mathbf{a}}^{\left(2\right)}={\mathbf{z}}^{\left(2\right)}$. This single value (0.6609) constitutes the compressed representation of our original 2D input, analogous to the projected value in PCA but learned through network parameters.

Step 3: Decoder Hidden Layer Computation: The decoder begins reconstructing the original input from the latent code:

$$\begin{array}{c}\hfill {\mathbf{z}}^{\left(3\right)}={\mathbf{W}}^{\left(3\right)}{\mathbf{a}}^{\left(2\right)}+{\mathbf{b}}^{\left(3\right)}=0.6\times 0.6609+0.3=0.6965,{\mathbf{a}}^{\left(3\right)}=\tanh \left({\mathbf{z}}^{\left(3\right)}\right)\approx 0.6029\end{array}$$

Step 4: Output Layer Computation: The final step produces the reconstructed output, which should ideally match the original input:

$$\begin{array}{c}\hfill {\mathbf{z}}^{\left(4\right)}={\mathbf{a}}^{\left(4\right)}={\widehat{\mathbf{x}}}_1={\mathbf{W}}^{\left(4\right)}{\mathbf{a}}^{\left(3\right)}+{\mathbf{b}}^{\left(4\right)}=\left[\genfrac{}{}{0ex}{}{0.7}{0.8}\right]\times 0.6029+\left[\genfrac{}{}{0ex}{}{0.4}{0.5}\right]=\left[\genfrac{}{}{0ex}{}{0.8220}{0.9823}\right]\end{array}$$

where the output ${\widehat{\mathbf{x}}}_1$ represents the network’s attempt to reconstruct the original input ${\mathbf{x}}_1$ from the compressed latent representation. From the above equation, it is clear that we are using the linear activation function in this layer.

Step 5: Loss Computation: To quantify how well the AE has reconstructed the input, we compute the MSE loss as follows:

$$\begin{array}{c}\hfill \mathcal{L}={\displaystyle \frac{1}{2}}{\parallel {\mathbf{x}}_1-{\widehat{\mathbf{x}}}_1\parallel }^2={\displaystyle \frac{1}{2}}\left[{\left(0.1780\right)}^2+{\left(2.0177\right)}^2\right]\approx 2.0514\end{array}$$

The factor ${\displaystyle \frac{1}{2}}$ simplifies gradient computation during backpropagation. The MSE loss of approximately 2.0514 quantifies the discrepancy between input and reconstruction, analogous to the reconstruction error in PCA but computed differently.

For reference, the complete forward transformation—from input to reconstruction—can be expressed compactly as follows:

$$\begin{array}{cc}\hfill {\mathbf{x}}_1& \stackrel{{\mathbf{W}}^{\left(1\right)}{\mathbf{x}}_1+{\mathbf{b}}^{\left(1\right)}}{\to }{\mathbf{z}}^{\left(1\right)}\stackrel{\tanh }{\to }{\mathbf{a}}^{\left(1\right)}\stackrel{{\mathbf{W}}^{\left(2\right)}{\mathbf{a}}^{\left(1\right)}+{\mathbf{b}}^{\left(2\right)}}{\to }{\mathbf{z}}^{\left(2\right)}={\mathbf{a}}^{\left(2\right)}\hfill \\ \hfill {\mathbf{a}}^{\left(2\right)}& \stackrel{{\mathbf{W}}^{\left(3\right)}{\mathbf{a}}^{\left(2\right)}+{\mathbf{b}}^{\left(3\right)}}{\to }{\mathbf{z}}^{\left(3\right)}\stackrel{\tanh }{\to }{\mathbf{a}}^{\left(3\right)}\stackrel{{\mathbf{W}}^{\left(4\right)}{\mathbf{a}}^{\left(3\right)}+{\mathbf{b}}^{\left(4\right)}}{\to }{\mathbf{z}}^{\left(4\right)}\stackrel{{\mathbf{z}}^{\left(4\right)}}{\to }{\widehat{\mathbf{x}}}_1\hfill \end{array}$$

(19)

4.4.2. Backpropagation: Gradient Computation

Backpropagation computes how each parameter contributed to the reconstruction error, allowing us to update the network to reduce future errors. This process applies the chain rule of calculus backward through the network, starting from the output layer. We denote the sensitivity of the loss $\mathcal{L}$ with respect to the pre-activation of layer l as follows:

$${\delta }^{\left(l\right)}\triangleq {\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{z}}^{\left(l\right)}}}.$$

We update all parameters using gradient descent with learning rate $\eta =0.01$. Each parameter moves in the direction opposite to its gradient, proportionally to the learning rate. For any layer with affine mapping

$${\mathbf{z}}^{\left(l\right)}={\mathbf{W}}^{\left(l\right)}{\mathbf{a}}^{(l-1)}+{\mathbf{b}}^{\left(l\right)}(\mathrm{where}{\mathbf{a}}^{\left(0\right)}\equiv {\mathbf{x}}_1\mathrm{for}l=1),$$

the gradients with respect to the weights and biases are

$${\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{W}}^{\left(l\right)}}}={\delta }^{\left(l\right)}{\left({\mathbf{a}}^{(l-1)}\right)}^{\top },{\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{b}}^{\left(l\right)}}}={\delta }^{\left(l\right)}.$$

Applying this to each layer,

$${\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{W}}^{\left(4\right)}}}={\delta }^{\left(4\right)}{\left({\mathbf{a}}^{\left(3\right)}\right)}^{\top },{\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{b}}^{\left(4\right)}}}={\delta }^{\left(4\right)},$$

$${\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{W}}^{\left(3\right)}}}={\delta }^{\left(3\right)}{\left({\mathbf{a}}^{\left(2\right)}\right)}^{\top },{\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{b}}^{\left(3\right)}}}={\delta }^{\left(3\right)},$$

$${\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{W}}^{\left(2\right)}}}={\delta }^{\left(2\right)}{\left({\mathbf{a}}^{\left(1\right)}\right)}^{\top },{\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{b}}^{\left(2\right)}}}={\delta }^{\left(2\right)},$$

$${\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{W}}^{\left(1\right)}}}={\delta }^{\left(1\right)}{\mathbf{x}}_1^{\top },{\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{b}}^{\left(1\right)}}}={\delta }^{\left(1\right)}.$$

We now follow this chain step by step, using the numerical values already computed in the forward pass.

Output Layer (Layer 4).

Forward: ${\widehat{\mathbf{x}}}_1={\mathbf{z}}^{\left(4\right)}$ (linear activation).

The gradient of the loss with respect to the output is the reconstruction error itself:

$${\delta }^{\left(4\right)}={\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{z}}^{\left(4\right)}}}={\widehat{\mathbf{x}}}_1-{\mathbf{x}}_1=\left[\begin{array}{c}-0.1780\\ -2.0177\end{array}\right].$$

Using this, we compute the gradients for ${\mathbf{W}}^{\left(4\right)}$ and ${\mathbf{b}}^{\left(4\right)}$:

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial L}{\partial {\mathbf{W}}^{\left(4\right)}}}& ={\delta }^{\left(4\right)}{\left({\mathbf{a}}^{\left(3\right)}\right)}^{\top }=\left[\genfrac{}{}{0ex}{}{-0.1780}{-2.0177}\right]\times 0.6029=\left[\genfrac{}{}{0ex}{}{-0.1073}{-1.2165}\right]\hfill \\ \hfill {\displaystyle \frac{\partial L}{\partial {\mathbf{b}}^{\left(4\right)}}}& ={\delta }^{\left(4\right)}=\left[\genfrac{}{}{0ex}{}{-0.1780}{-0.1780}\right].\hfill \end{array}$$

Decoder Hidden Layer (Layer 3).

Forward: ${\mathbf{z}}^{\left(4\right)}={\mathbf{W}}^{\left(4\right)}{\mathbf{a}}^{\left(3\right)}+{\mathbf{b}}^{\left(4\right)}$. Thus,

$${\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{a}}^{\left(3\right)}}}={\left({\mathbf{W}}^{\left(4\right)}\right)}^{\top }{\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{z}}^{\left(4\right)}}}={\left({\mathbf{W}}^{\left(4\right)}\right)}^{\top }{\delta }^{\left(4\right)}=\left[\begin{array}{cc}0.7& 0.8\end{array}\right]\left[\begin{array}{c}-0.1780\\ -2.0177\end{array}\right]=-1.7388.$$

The activation function is ${\mathbf{a}}^{\left(3\right)}=\tanh \left({\mathbf{z}}^{\left(3\right)}\right)$, whose derivative is

$${\tanh }^{\prime }\left({\mathbf{z}}^{\left(3\right)}\right)=1-{\tanh }^2\left({\mathbf{z}}^{\left(3\right)}\right)=1-{\left({\mathbf{a}}^{\left(3\right)}\right)}^2\left(\mathrm{element}\text{-}\mathrm{wise}\right).$$

With ${\mathbf{a}}^{\left(3\right)}=0.6029$, we obtain ${\tanh }^{\prime }\left({\mathbf{z}}^{\left(3\right)}\right)=1-{0.6029}^2=0.6365.$

Therefore,

$${\delta }^{\left(3\right)}={\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{z}}^{\left(3\right)}}}={\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{a}}^{\left(3\right)}}}\odot {\tanh }^{\prime }\left({\mathbf{z}}^{\left(3\right)}\right)=(-1.7388)\times 0.6365\approx -1.1065.$$

From this, we obtain the parameter gradients:

$${\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{W}}^{\left(3\right)}}}={\delta }^{\left(3\right)}{\left({\mathbf{a}}^{\left(2\right)}\right)}^{\top }=-1.1065\times 0.6609\approx -0.7310,{\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{b}}^{\left(3\right)}}}={\delta }^{\left(3\right)}=-1.1065.$$

Similarly, all the other gradients of the other earlier layers will be calculated. With all gradients computed, we now have a complete picture of how each parameter contributes to the reconstruction error.

From the above calculations, the output layer parameters update will be as follows:

$$\begin{array}{cc}\hfill {\mathbf{W}}_{\mathrm{new}}^{\left(4\right)}& ={\mathbf{W}}^{\left(4\right)}-\eta {\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{W}}^{\left(4\right)}}}=\left[\genfrac{}{}{0ex}{}{0.7}{0.8}\right]-0.01\times \left[\genfrac{}{}{0ex}{}{-0.1073}{-1.2165}\right]=\left[\genfrac{}{}{0ex}{}{0.7011}{0.8122}\right]\hfill \\ \hfill {\mathbf{b}}_{\mathrm{new}}^{\left(4\right)}& ={\mathbf{b}}^{\left(4\right)}-\eta {\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{b}}^{\left(4\right)}}}=\left[\genfrac{}{}{0ex}{}{0.4}{0.5}\right]-0.01\times \left[\genfrac{}{}{0ex}{}{-0.1780}{-2.0177}\right]=\left[\genfrac{}{}{0ex}{}{0.4018}{0.5202}\right]\hfill \end{array}$$

Similarly, all the other weights and biases will be calculated for the other layers to be ${\mathbf{W}}^{\left(3\right)}\approx 0.6073$, ${\mathbf{b}}^{\left(3\right)}\approx 0.3111$, ${\mathbf{W}}^{\left(2\right)}\approx 0.5061$, ${\mathbf{b}}^{\left(2\right)}\approx 0.2066$, ${\mathbf{W}}^{\left(1\right)}\approx \left[\begin{array}{cc}0.3005& 0.4015\end{array}\right]$, ${\mathbf{b}}^{\left(1\right)}\approx 0.1005$. After this single update, all parameters have been adjusted slightly to reduce the reconstruction error for this training sample.

4.5. The Fundamental Connection: When Autoencoders Generalize PCA

The relationship between Autoencoders and PCA is not merely conceptual but mathematically precise under specific conditions. This connection provides critical insight into how neural networks extend and generalize classical linear methods.

Consider an Autoencoder with the following specific characteristics:

1.

Linear Activations: All activation functions are identity functions.

2.

Single Hidden Layer: One bottleneck layer of dimension $K<M$, where M is the input dimension.

3.

Mean Squared Error: The loss function is the standard MSE: $\mathcal{L}={\displaystyle \frac{1}{2}}{\parallel \mathbf{x}-\widehat{\mathbf{x}}\parallel }^2$.

4.

No Regularization: No weight decay or other regularization terms are applied.

Under these conditions, the Autoencoder learns an optimal solution that spans the same subspace as PCA. Specifically, let the encoder and decoder be defined by weight matrices ${\mathbf{W}}^{\left(E\right)}\in {\mathbb{R}}^{K\times M}$ and ${\mathbf{W}}^{\left(D\right)}\in {\mathbb{R}}^{M\times K}$, with biases ${\mathbf{b}}^{\left(E\right)}$ and ${\mathbf{b}}^{\left(D\right)}$. The network computes

$$\mathbf{z}={\mathbf{W}}^{\left(E\right)}\mathbf{x}+{\mathbf{b}}^{\left(E\right)},\widehat{\mathbf{x}}={\mathbf{W}}^{\left(D\right)}\mathbf{z}+{\mathbf{b}}^{\left(D\right)}={\mathbf{W}}^{\left(D\right)}({\mathbf{W}}^{\left(E\right)}\mathbf{x}+{\mathbf{b}}^{\left(E\right)})+{\mathbf{b}}^{\left(D\right)}$$

The optimization problem is to minimize the reconstruction error:

$$\underset{{\mathbf{W}}^{\left(E\right)},{\mathbf{W}}^{\left(D\right)},{\mathbf{b}}^{\left(E\right)},{\mathbf{b}}^{\left(D\right)}}{min}\mathbb{E}[\parallel \mathbf{x}-{\mathbf{W}}^{\left(D\right)}{\mathbf{W}}^{\left(E\right)}\mathbf{x}-({\mathbf{W}}^{\left(D\right)}{\mathbf{b}}^{\left(E\right)}+{\mathbf{b}}^{\left(D\right)}){\parallel }^2]$$

To understand the optimal solution, it is helpful to reframe the problem. Define the combined transformation matrix $\mathbf{C}={\mathbf{W}}^{\left(D\right)}{\mathbf{W}}^{\left(E\right)}\in {\mathbb{R}}^{M\times M}$ and the combined bias vector $\mathbf{d}={\mathbf{W}}^{\left(D\right)}{\mathbf{b}}^{\left(E\right)}+{\mathbf{b}}^{\left(D\right)}$. The reconstruction becomes $\widehat{\mathbf{x}}=\mathbf{C}\mathbf{x}+\mathbf{d}$, and the objective simplifies to minimizing $\mathbb{E}\left[\right|\mathbf{x}-(\mathbf{C}\mathbf{x}+\mathbf{d}){|}^2]$.

The math shows that the optimal bias basically centers the data. By setting the derivative with respect to $\mathbf{d}$ to zero, one finds that the optimal bias acts like the mean-centering step in PCA. If we center the data beforehand ($\tilde{\mathbf{x}}=\mathbf{x}-\mu$), the biases become ${\mathbf{b}}^{\left(E\right)}=\mathbf{0}$ and ${\mathbf{b}}^{\left(D\right)}=\mathbf{0}$, and the problem reduces to

$$\underset{{\mathbf{W}}^{\left(E\right)},{\mathbf{W}}^{\left(D\right)}}{min}\mathbb{E}\left[\parallel \tilde{\mathbf{x}}-{\mathbf{W}}^{\left(D\right)}{\mathbf{W}}^{\left(E\right)}\tilde{\mathbf{x}}{\parallel }^2\right].$$

Here, the product $\mathbf{C}={\mathbf{W}}^{\left(D\right)}{\mathbf{W}}^{\left(E\right)}$ is a matrix of at most rank K, because ${\mathbf{W}}^{\left(E\right)}$ projects from M to K dimensions. The optimization problem is therefore to find the optimal rank-K linear approximation to the centered data. This is precisely the problem solved by PCA. More specifically, at the global optimum, the columns of ${\mathbf{W}}^{\left(D\right)}$ will cover the same space spanned by the first K principal components of the data covariance matrix. Furthermore, ${\mathbf{W}}^{\left(E\right)}={\left({\mathbf{W}}^{\left(D\right)}\right)}^{\top }$, making the encoder perform projection onto this space, and the latent representation $\mathbf{z}$ contains the first K principal component scores of the centered input. This means that a linear Autoencoder with MSE loss performs exactly the same dimensionality reduction as PCA: it finds the K-dimensional linear subspace that minimizes the squared reconstruction error. This equivalence, first proven by [34], reveals that PCA is actually just a special case of the Autoencoder framework. The only difference is that while PCA gives a unique set of eigenvectors, the Autoencoder might find different weights that cover the same space, depending on how the training starts [35,36].

4.6. Variants of Autoencoders

While the standard Autoencoder (Section 4) provides a flexible framework for unsupervised representation learning, numerous architectural variants have emerged to address specific limitations or application requirements. These variants typically modify one or more aspects of the basic AE: the architecture (e.g., replacing fully connected layers), the loss function (adding regularization terms), or the training objective (e.g., robustness to noise). Each variant tailors the AE to particular data modalities or desired representation properties. Notably, the VAE represents a foundational probabilistic extension that will be thoroughly examined in Section 5. More details and other variants can be found in [37].

4.6.1. Regularized Autoencoders

These versions add an extra “penalty” to the loss function to give the latent space specific shapes.

Sparse Autoencoder (SAE): This version uses a “less is more” approach. It forces most of the neurons in the latent space to stay at zero (inactive). This forces the network to find the most unique and important features that can represent the data with only a few active neurons [38]. This helps the network ignore small noise and focus on the real structure.

Contractive Autoencoder (CAE): This version makes the network “stiff” or robust. It adds a penalty that ensures that if you change the input just a little bit, the latent code does not change much [39].

Laplacian Autoencoder (LAE): This version focuses on “neighborhoods.” It tries to make sure that if two data points were neighbors in the original space, they stay neighbors in the latent space [40].

4.6.2. Architectural Variants for Specific Data Types

Convolutional Autoencoder (CAE): Instead of standard connections, this version uses “filters” (convolutional layers). This is the gold standard for image data because it understands that pixels near each other are related [41].

Bayesian and Diffusion Autoencoders: These are advanced versions that deal with uncertainty and high-quality generation. A Bayesian AE uses probability for its weights, while a Diffusion AE learns by “cleaning” noisy data [42].

Diffusion Autoencoder: This integrates the AE framework with diffusion models, where the encoder maps an input to a latent code, and a diffusion-based decoder learns to reconstruct the input by progressively denoising. The training involves optimizing a diffusion loss alongside reconstruction terms [43].

Other notable variants include Robust Autoencoders (designed to handle noisy inputs), Recurrent Autoencoders (using RNNs/LSTMs for sequential data), and Graph Autoencoders (operating on graph-structured data). Each of these versions shows how we can adapt the simple idea of “compress and reconstruct” to work for almost any kind of data.

5. Variational Autoencoder (VAE)

While Autoencoders excel at learning compressed, non-linear representations, their deterministic nature and unstructured latent space, as discussed in Section 4.2, fundamentally limit their utility as generative models. The Variational Autoencoder, introduced by Kingma and Welling (2013), addresses this core limitation by re-imagining the Autoencoder within a probabilistic framework [5]. Think of it this way: A standard Autoencoder maps an input to a single, fixed point in the latent space. The VAE’s “Variational” twist is that it maps an input to a fuzzy “region” or a probability distribution over that space.

The VAE accomplishes this by combining three foundational principles: (1) probabilistic models that define a generative process, (2) variational inference for approximating complex math, and (3) deep neural networks for learning and parameterizing complex, non-linear relationships. The VAE is not just a simple encoder–decoder architecture; it is a deep latent variable model designed to learn the underlying probability distribution $p\left(\mathbf{x}\right)$ of the data.

5.1. Architectural Components: A Probabilistic Reinterpretation

The VAE retains the encoder–decoder structure but redefines their functions. We represent all learnable parameters (weights and biases) as $\mathrm{\Theta }=\{\varphi ,\theta \}$, the encoder parameters as $\varphi$ and the decoder parameters as $\theta$.

1.

Probabilistic Encoder (Recognition Model): The encoder is a neural network parameterized by $\varphi$. It serves as an approximate variational inference model, meaning that it uses a single set of parameters $\varphi$ to approximate the posterior for any input data point, enabling efficient learning on large datasets. Its input is a data point $\mathbf{x}$. Its outputs are the parameters of the approximate posterior distribution $q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})$ (see Figure 8). For a standard VAE, we assume that this distribution is a multivariate Gaussian with a diagonal covariance matrix. For a latent space of dimension K, the encoder’s output for a given input $\mathbf{x}$ consists of two K-dimensional vectors:

$$\begin{array}{cc}\hfill \mu & =f_{\varphi }^{\mu }\left(\mathbf{x}\right),\mu \in {\mathbb{R}}^K\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill \log {\sigma }^2& =f_{\varphi }^{\sigma }\left(\mathbf{x}\right),{\sigma }^2\in {\mathbb{R}}^K\hfill \end{array}$$

(21)

where $f_{\varphi }^{\mu }$ and $f_{\varphi }^{\sigma }$ are the output layers (or branches) of the encoder network, producing the parameters that define the distribution $q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})=\mathcal{N}(\mathbf{z};\mu ,\mathrm{diag}\left({\sigma }^2\right)).$ Here, $\mu$ contains the means and ${\sigma }^2$ contains the variances for each of the K latent dimensions. We use log-variance for numerical stability. This formulation defines a K-dimensional multivariate Gaussian distribution with a diagonal covariance matrix $\mathrm{diag}\left({\sigma }^2\right)$, indicating that each latent dimension $z_k$ is modeled as an independent Gaussian with its own mean ${\mu }_k$ and variance ${\sigma }_k^2$.

In the VAE’s probabilistic framework, $\varphi$ represents the weights of the encoder (the inference network). It tries to approximate the true, intractable posterior $p_{\theta }\left(\mathbf{z}\right|\mathbf{x})$ by varying the parameters of a simpler distribution (a Gaussian) until it is as close as possible—this is the essence of variational inference.

For example, if $K=2$, the encoder for an input ${\mathbf{x}}_1$ outputs ${\mu }_1={[{\mu }_{1,1},{\mu }_{1,2}]}^{\top }$ and ${\sigma }_1^2={[{\sigma }_{1,1}^2,{\sigma }_{1,2}^2]}^{\top }$. This means the latent code ${\mathbf{z}}_1$ is assumed to be drawn from a distribution where the first dimension has mean ${\mu }_{1,1}$ and variance ${\sigma }_{1,1}^2$, and the second dimension has mean ${\mu }_{1,2}$ and variance ${\sigma }_{1,2}^2$, with the dimensions being independent.

2.

Latent Space Sampling: To obtain a latent vector for reconstruction or generation, we sample from the inferred distribution: $\mathbf{z}\sim q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})$. However, the sampling operation ′∼′ is a random process and its derivative is undefined; it “blocks gradient flow,” meaning that we cannot calculate gradients with respect to the encoder parameters $\varphi$ through this operation, which prevents training via backpropagation.

The reparameterization trick provides an elegant solution. Instead of sampling $\mathbf{z}$ directly, we express it as a deterministic, differentiable function of the encoder’s outputs and an auxiliary noise variable. We first sample a noise vector $ϵ\sim \mathcal{N}(\mathbf{0},\mathbf{I})$ from a standard normal distribution. Then, we compute the latent code as follows:

$$\begin{array}{c}\hfill \mathbf{z}=\mu +\sigma \odot ϵ,\end{array}$$

where ⊙ denotes element-wise multiplication and $\sigma =\sqrt{{\sigma }^2}$. Crucially, all randomness is now contained in $ϵ$, which is independent of $\varphi$. The path from $\mu$ and $\sigma$ (which depend on $\varphi$) to $\mathbf{z}$ is now fully differentiable, allowing gradients to flow back to the encoder.

Continuing the $K=2$ example, we sample $ϵ={[{ϵ}_1,{ϵ}_2]}^{\top }$ from $\mathcal{N}(\mathbf{0},\mathbf{I})$ and compute ${\mathbf{z}}_1={[{\mu }_{1,1}+{\sigma }_{1,1}·{ϵ}_1,{\mu }_{1,2}+{\sigma }_{1,2}·{ϵ}_2]}^{\top }$. The distribution $q_{\varphi }\left(\mathbf{z}\right|{\mathbf{x}}_1)$ describes the probability density over all possible $\mathbf{z}$ given ${\mathbf{x}}_1$; ${\mathbf{z}}_1$ is one specific sample from this distribution.

3.

Probabilistic Decoder (Generative Model): The decoder, parameterized by $\theta$, is reinterpreted as defining a likelihood distribution $p_{\theta }\left(\mathbf{x}\right|\mathbf{z})$ over the input space. This represents the conditional likelihood in the Bayesian network. In the probabilistic framework, $\theta$ represents the weights of the decoder (the generative network). Its input is a latent sample $\mathbf{z}$ (obtained via the reparameterization trick during training, or from the prior $p\left(\mathbf{z}\right)=\mathcal{N}(\mathbf{0},\mathbf{I})$ during generation). The decoder neural network typically outputs the parameters of a distribution in data space. For real-valued data (e.g., images normalized to [0, 1]), a common choice is a Gaussian distribution with identity covariance, where the decoder’s output $g_{\theta }\left(\mathbf{z}\right)$ is interpreted as the mean of the reconstructed data:

$$\begin{array}{c}\hfill p_{\theta }\left(\mathbf{x}\right|\mathbf{z})=\mathcal{N}\left(\mathbf{x};g_{\theta }\left(\mathbf{z}\right),\mathbf{I}\right).\end{array}$$

Thus, for our sampled ${\mathbf{z}}_1$, the decoder outputs $g_{\theta }\left({\mathbf{z}}_1\right)$, which is the mean of the Gaussian distribution from which the reconstructed data ${\widehat{\mathbf{x}}}_1$ is likely drawn. The decoder learns to map any latent point $\mathbf{z}$ to a distribution over possible data points $\mathbf{x}$.

To train this probabilistic model, we need an objective function that serves two distinct yet complementary purposes. First, the decoder must learn to accurately reconstruct the input data from the latent code, similar to a standard autoencoder. Second, the encoder’s learned posterior distributions must be regularized to match a simple prior distribution, typically chosen as a standard normal distribution $p\left(\mathbf{z}\right)=\mathcal{N}(\mathbf{0},\mathbf{I})$. This prior acts as a template, encouraging the latent codes for all data points to be distributed in a well-behaved, continuous region. Without this, the encoder might learn to map different inputs to disjoint, tightly clustered latent distributions, creating “holes” in the latent space from which the decoder cannot generate coherent outputs. To clarify these important terms

• The prior $p\left(\mathbf{z}\right)$ is what we believe about the latent space before seeing any data: a “clean map” (standard normal).
• The true posterior $p_{\theta }\left(\mathbf{z}\right|\mathbf{x})$ is what we would infer after seeing a specific input, but it is mathematically intractable to compute.
• The approximate posterior $q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})$ is what the encoder actually produces: a “messy, specific region” of the map for that input.

The training goal is to make this approximation both accurate (close to the true posterior) and well-structured (aligned with the prior).

5.2. The Core Challenge and the Variational Inference Solution

At its core, training a VAE means teaching it to model the probability distribution of our data. For any given data point $\mathbf{x}$ (like an image), we want our model parameters $\theta$—which primarily govern the decoder—to assign a high probability to that data. This probability is expressed as $p_{\theta }\left(\mathbf{x}\right)$, and our goal is to maximize it across our entire dataset.

The challenge comes from how we calculate $p_{\theta }\left(\mathbf{x}\right)$. Since $\mathbf{x}$ is generated from the latent variable $\mathbf{z}$, we must consider all possible $\mathbf{z}$ that could have produced it. Mathematically, we integrate over the latent space:

$$p_{\theta }\left(\mathbf{x}\right)=\int p_{\theta }(\mathbf{x},\mathbf{z})d\mathbf{z}=\int p_{\theta }\left(\mathbf{x}\right|\mathbf{z})p\left(\mathbf{z}\right)d\mathbf{z}.$$

This is the fundamental problem: this integral is intractable—meaning that it is impossible to compute exactly for complex models like neural networks. This is because of the following:

1.

We need to consider all possible $\mathbf{z}$ in a high-dimensional space.

2.

Only a tiny fraction of $\mathbf{z}$ values produce a plausible $\mathbf{x}$, making random sampling hopelessly inefficient.

3.

We cannot calculate the true posterior $p_{\theta }\left(\mathbf{z}\right|\mathbf{x})={\displaystyle \frac{p_{\theta }\left(\mathbf{x}\right|\mathbf{z})p\left(\mathbf{z}\right)}{p_{\theta }\left(\mathbf{x}\right)}}$, because it requires the very term $p_{\theta }\left(\mathbf{x}\right)$ that we are trying to compute.

The term “Variational” refers to Variational Inference. In statistics, when a distribution is too complex to calculate, we pick a family of simpler distributions (like Gaussians) and “vary” their parameters (mean and variance) until they look as much like the complex one as possible. The VAE uses a neural network to do that “varying” automatically. In VAE, instead of computing the intractable $p_{\theta }\left(\mathbf{x}\right)$ (complex distribution) directly, we approximate the true posterior $p_{\theta }\left(\mathbf{z}\right|\mathbf{x})$ with a simpler, parameterized distribution $q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})$—our encoder. We then construct a lower bound on $\log p_{\theta }\left(\mathbf{x}\right)$ that we can actually compute and maximize.

Here’s how we derive this bound step by step:

1.

We work with $\log p_{\theta }\left(\mathbf{x}\right)$ instead of $p_{\theta }\left(\mathbf{x}\right)$ because probabilities multiply but log-probabilities add, making the math tractable and preventing numerical underflow.

2.

We introduce the expectation under $q_{\varphi }$ because we need to bring the encoder into the equation to connect it to the parameters we want to optimize ($\varphi$).

$$\begin{array}{c}\hfill \log p_{\theta }\left(\mathbf{x}\right)={\mathbb{E}}_{q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})}\left[\log p_{\theta }\left(\mathbf{x}\right)\right].\end{array}$$

Since $\log p_{\theta }\left(\mathbf{x}\right)$ is constant with respect to $\mathbf{z}$, taking its expectation does not change its value.

3.

Expand using Bayes’ rule: We rewrite $p_{\theta }\left(\mathbf{x}\right)$ using the joint probability and the posterior:

$$\log p_{\theta }\left(\mathbf{x}\right)={\mathbb{E}}_{q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})}\left[\log {\displaystyle \frac{p_{\theta }(\mathbf{x},\mathbf{z})}{p_{\theta }\left(\mathbf{z}\right|\mathbf{x})}}\right].$$

4.

Introduce our encoder $q_{\varphi }$ strategically: We multiply and divide by $q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})$ inside the expectation to prepare for splitting the expression:

$$\begin{array}{c}\hfill \log p_{\theta }\left(\mathbf{x}\right)={\mathbb{E}}_{q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})}\left[\log \left({\displaystyle \frac{p_{\theta }(\mathbf{x},\mathbf{z})}{q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})}}·{\displaystyle \frac{q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})}{p_{\theta }\left(\mathbf{z}\right|\mathbf{x})}}\right)\right].\end{array}$$

5.

Split the logarithm and recognize the KL divergence: Using the property $\log \left(ab\right)=\log a+\log b$, we get:

$$\log p_{\theta }\left(\mathbf{x}\right)=\underset{\mathrm{ELBO}\mathcal{L}(\theta ,\varphi ;\mathbf{x})}{\underbrace{{\mathbb{E}}_{q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})}\left[\log {\displaystyle \frac{p_{\theta }(\mathbf{x},\mathbf{z})}{q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})}}\right]}}+\underset{\ge 0}{\underbrace{D_{KL}(q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})\parallel p_{\theta }\left(\mathbf{z}\right|\mathbf{x}))}}.$$

This decomposition reveals that the KL divergence $D_{\mathrm{KL}}(q_{\varphi }\parallel p_{\theta })$ is always non-negative (it’s 0 only if $q_{\varphi }$ equals $p_{\theta }$ exactly), we have:

$$\log p_{\theta }\left(\mathbf{x}\right)\ge \mathcal{L}(\theta ,\varphi ;\mathbf{x}).$$

$\mathcal{L}$ is called the Evidence Lower BOund (ELBO). We can rewrite $\mathcal{L}(\theta ,\varphi ;\mathbf{x})$ by expanding the joint probability $p_{\theta }(\mathbf{x},\mathbf{z})=p_{\theta }\left(\mathbf{x}\right|\mathbf{z})p\left(\mathbf{z}\right)$:

$$\mathcal{L}(\theta ,\varphi ;\mathbf{x})={\mathbb{E}}_{q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})}\left[\log p_{\theta }\left(\mathbf{x}\right|\mathbf{z})+\log p\left(\mathbf{z}\right)-\log q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})\right].$$

Separating the terms gives us:

$$\mathcal{L}(\theta ,\varphi ;\mathbf{x})={\mathbb{E}}_{q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})}\left[\log p_{\theta }\left(\mathbf{x}\right|\mathbf{z})\right]+{\mathbb{E}}_{q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})}\left[\log p\left(\mathbf{z}\right)-\log q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})\right].$$

Notice that the second term is exactly the negative KL divergence:

$${\mathbb{E}}_{q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})}\left[\log p\left(\mathbf{z}\right)-\log q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})\right]=-D_{KL}(q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})\parallel p\left(\mathbf{z}\right)).$$

Therefore, we arrive at the final, interpretable form of the ELBO:

$$\begin{array}{c}\hfill \mathcal{L}(\theta ,\varphi ;\mathbf{x})=\underset{\mathrm{Reconstruction}\mathrm{Term}}{\underbrace{{\mathbb{E}}_{q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})}\left[\log p_{\theta }\left(\mathbf{x}\right|\mathbf{z})\right]}}-\underset{\mathrm{Regularization}\mathrm{Term}}{\underbrace{D_{KL}(q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})\parallel p\left(\mathbf{z}\right))}}=L_{ELBO}(\mathbf{x};\varphi ,\theta ).\end{array}$$

(22)

By maximizing this lower bound $\mathcal{L}$, we achieve two objectives:

• Increasing the Likelihood: We push up the value of $\log p_{\theta }\left(\mathbf{x}\right)$, which is our original goal.
• Aligning Distributions: We simultaneously minimize the difference between our encoder $q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})$ and the true posterior distribution.

The ELBO naturally decomposes into the two components that we intuitively need:

• Reconstruction Loss: ${\mathbb{E}}_{q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})}\left[\log p_{\theta }\left(\mathbf{x}\right|\mathbf{z})\right]$, which for a Gaussian decoder becomes the negative MSE (equivalent to maximizing log-likelihood).
• Regularization Loss: $D_{\mathrm{KL}}\left(q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})\right|p\left(\mathbf{z}\right))$. This KL divergence has a closed-form solution when both distributions are Gaussian, penalizing deviations of the encoder’s output from the standard normal prior $\mathcal{N}(\mathbf{0},\mathbf{I})$.

Training a VAE is a balancing act between these two goals. The reconstruction loss asks “Does the output look like the input?” The regularization loss (KL divergence) forces the latent space distributions to look like a standard normal distribution. Without this regularizer, the network would give every input its own isolated “island” in latent space, preventing smooth interpolation and generation of new data.

5.3. The Variational Lower Bound (ELBO)

For practical implementation, we minimize the negative ELBO (see Equation 22) rather than maximizing the bound. The training procedure aims to maximize this bound, thereby approximately maximizing the data log-likelihood. For a dataset of N points, we sum the individual bounds:

$$\underset{\varphi ,\theta }{max}\sum _{i=1}^N{L}_{ELBO}({\mathbf{x}}_i;\varphi ,\theta ).$$

In practice, we equivalently minimize the negative ELBO, ${\mathcal{L}}_{\mathrm{VAE}}(\varphi ,\theta ;\mathbf{x})=-{\mathcal{L}}_{ELBO}(\mathbf{x};\varphi ,\theta )$:

$${\mathcal{L}}_{\mathrm{VAE}}(\varphi ,\theta ;\mathbf{x})=-{\mathbb{E}}_{q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})}\left[\log p_{\theta }\left(\mathbf{x}\right|\mathbf{z})\right]+D_{KL}(q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})\Vert p\left(\mathbf{z}\right)).$$

which serves as our loss function for gradient descent. Let’s examine its two critical components:

• Reconstruction Term (${\mathbb{E}}_{q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})}\left[\log p_{\theta }\left(\mathbf{x}\right|\mathbf{z})\right]$): This term measures how well the model can reconstruct the input $\mathbf{x}$ from a latent code $\mathbf{z}$ sampled from the encoder’s posterior. It is the expected log-likelihood of $\mathbf{x}$ under the decoder’s distribution. Maximizing this term forces the decoder to produce outputs $g_{\theta }\left(\mathbf{z}\right)$ that are likely given the true data. For a Gaussian decoder $p_{\theta }\left(\mathbf{x}\right|\mathbf{z})=\mathcal{N}(\mathbf{x};g_{\theta }\left(\mathbf{z}\right),\mathbf{I})$, maximizing the log-likelihood is equivalent to minimizing the MSE between the original input $\mathbf{x}$ and the reconstruction $\widehat{\mathbf{x}}=g_{\theta }\left(\mathbf{z}\right)$. The expectation is approximated during training using the reparameterized latent sample $\mathbf{z}=\mu +\sigma \odot ϵ$.
• Regularization Term ($D_{KL}(q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})\Vert p\left(\mathbf{z}\right))$): This is the Kullback–Leibler (KL) divergence between the encoder’s posterior $q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})$ and the prior $p\left(\mathbf{z}\right)=\mathcal{N}(\mathbf{0},\mathbf{I})$. It acts as a regularizer on the latent space. Minimizing this term (as we subtract it in the ELBO) pushes the distribution $q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})$ for each data point to be close to the standard normal prior. This has several crucial effects: it prevents the posterior distributions from collapsing into isolated point masses (overfitting), encourages smoothness and continuity in the latent space, and ensures that, for generation, sampling a random $\mathbf{z}\sim p\left(\mathbf{z}\right)$ will land in a region of the latent space that the decoder understands. The KL divergence for Gaussian distributions has a convenient closed-form expression, making computation efficient.

Therefore, the total VAE loss for a single data point is computed and backpropagated as follows:

1.

The input ${\mathbf{x}}_1$ is passed through the encoder to get ${\mu }_1$ and ${\sigma }_1^2$.

2.

A latent code is sampled via reparameterization:

$${\mathbf{z}}_1={\mu }_1+{\sigma }_1\odot ϵ.$$

3.

The decoder maps ${\mathbf{z}}_1$ to the reconstruction parameters (e.g., the mean ${\widehat{\mathbf{x}}}_1=g_{\theta }\left({\mathbf{z}}_1\right)$).

4.

The loss ${\mathcal{L}}_{\mathrm{VAE}}$ is computed as follows:

$${\mathcal{L}}_{\mathrm{VAE}}=\underset{\mathrm{Reconstruction}\mathrm{Loss}}{\underbrace{\mathrm{MSE}({\mathbf{x}}_1,{\widehat{\mathbf{x}}}_1)}}+\underset{\mathrm{KL}\mathrm{Divergence}(\mathrm{Closed}\mathrm{Form})}{\underbrace{{\displaystyle \frac{1}{2}}\sum _{k=1}^K\left({\mu }_{1,k}^2+{\sigma }_{1,k}^2-\log \left({\sigma }_{1,k}^2\right)-1\right)}}.$$

As we see above, in practice, the expectation is approximated using a single sample of $\mathbf{z}$, and the KL divergence for a Gaussian prior and posterior has a convenient closed-form expression.

The total loss is backpropagated to update both the encoder parameters ($\varphi$) and decoder parameters ($\theta$). Gradients of this loss with respect to these parameters are calculated and used for model updates.

In summary, the VAE employs variational inference to transform an intractable maximum likelihood problem into the optimization of the ELBO (Evidence Lower Bound). This differentiable objective balances the dual goals of accurate data reconstruction and a structured, generative latent space, enabling the VAE to learn a powerful and useful model of the data distribution.

5.4. From Deterministic to Probabilistic Latent Spaces

The critical conceptual leap from the AE to the VAE is the treatment of the latent space. In a standard AE, the encoder outputs a single, deterministic latent vector $\mathbf{z}$ for each input $\mathbf{x}$. In the VAE, the encoder maps an input $\mathbf{x}$ to a probability distribution over the latent space, typically modeled as a multivariate Gaussian characterized by a mean vector $\mu$ and a variance vector ${\sigma }^2$.

This probabilistic formulation introduces powerful consequences: it acts as a regularization mechanism, as the loss function includes the KL divergence term that pushes the learned distribution $q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})$ toward a standard normal distribution $\mathcal{N}(0,\mathbf{I})$. This constraint creates a continuous, well-structured latent space where interpolation between points corresponds to meaningful changes in the decoded output. Second, by sampling from this prior and passing the sample through the trained decoder, the VAE can generate novel, realistic data points that resemble the training data.

5.5. Key Differences from the Standard Autoencoder

• Latent Representation: The AE produces a deterministic point $\mathbf{z}$. The VAE produces a probabilistic distribution $q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})$, from which $\mathbf{z}$ is sampled.
• Training Objective: The AE minimizes a simple reconstruction loss. The VAE maximizes the ELBO, which balances reconstruction accuracy and latent space regularization.
• Latent Space Structure: The AE’s latent space is unconstrained and can be discontinuous. The VAE’s latent space is explicitly regularized to be continuous and smooth, converging toward a known prior distribution. This structural difference is what unlocks the VAE’s generative power.
• Capability: AEs are primarily compression and representation learning models. VAEs are, by design, generative models capable of creating new data samples.

5.6. Numerical Example

Building on the AE example in Section 4.4, we now trace a VAE forward and backward pass using the same input ${\mathbf{x}}_1=\left[\begin{array}{c}1.00\\ 3.00\end{array}\right]$ and a similar architecture. The key difference is that the VAE encoder has two output branches—one for the mean $\mu$ and one for the log-variance $\log {\sigma }^2$—and uses the reparameterization trick to sample $\mathbf{z}$. The network architecture of the VAE will be: Input Layer: 2 neurons, Encoder Hidden Layer: 1 neuron with tanh activation, Bottleneck Distribution Parameters: (i) Mean ($\mu$): 1 linear neuron, and (ii) Log-Variance ($\log {\sigma }^2$): 1 linear neuron, Sampling Layer: Uses the reparameterization trick: $\mathbf{z}=\mu +\sigma \odot ϵ$, Decoder Hidden Layer: 1 neuron with tanh activation, and Output Layer: 2 linear neurons (reconstruction).

For clarity, we initialize all parameters with simple values. For the encoder pathway (to $\mu$),

$$\begin{array}{c}\hfill {\mathbf{W}}_{\mu }^{\left(1\right)}=[0.30.4],{\mathbf{b}}_{\mu }^{\left(1\right)}=0.1,{\mathbf{W}}_{\mu }^{\left(2\right)}=\left[0.5\right],{\mathbf{b}}_{\mu }^{\left(2\right)}=0.2\end{array}$$

where the encoder pathway (to $\log {\sigma }^2$) will be:

$$\begin{array}{cc}\hfill {\mathbf{W}}_{\sigma }^{\left(1\right)}& =[0.250.35],{\mathbf{b}}_{\sigma }^{\left(1\right)}=0.15,{\mathbf{W}}_{\sigma }^{\left(2\right)}=\left[\begin{array}{c}0.45\end{array}\right],{\mathbf{b}}_{\sigma }^{\left(2\right)}=0.25\hfill \end{array}$$

and the decoder pathway will be

$$\begin{array}{c}{\mathbf{W}}^{\left(3\right)}=\left[0.6\right],{\mathbf{b}}^{\left(3\right)}=0.3,{\mathbf{W}}^{\left(4\right)}=[0.70.8],{\mathbf{b}}^{\left(4\right)}=\left[0.4,,0.5\right]\end{array}$$

Note on Reparameterization: For reproducibility, we fix $ϵ=0.5$ (normally $ϵ\sim \mathcal{N}(\mathbf{0},\mathbf{I})$).

5.6.1. Forward Propagation

Step 1: Encoder Hidden Layer Computation: The input first passes through the shared encoder hidden layer (identical computation to AE). First, the path to $\mu$ will be:

$$\begin{array}{c}\hfill {\mathbf{z}}_{\mu }^{\left(1\right)}={\mathbf{W}}_{\mu }^{\left(1\right)}{\mathbf{x}}_1+{\mathbf{b}}_{\mu }^{\left(1\right)}=[0.30.4]\left[\genfrac{}{}{0ex}{}{1.00}{3.00}\right]+0.1=1.6,{\mathbf{a}}_{\mu }^{\left(1\right)}=\tanh \left({\mathbf{z}}_{\mu }^{\left(1\right)}\right)\approx 0.9217\end{array}$$

and the path to $\log {\sigma }^2$ will be:

$$\begin{array}{c}\hfill {\mathbf{z}}_{\sigma }^{\left(1\right)}={\mathbf{W}}_{\sigma }^{\left(1\right)}{\mathbf{x}}_1+{\mathbf{b}}_{\sigma }^{\left(1\right)}=[0.250.35]\left[\genfrac{}{}{0ex}{}{1.00}{3.00}\right]+0.15=1.45,{\mathbf{a}}_{\sigma }^{\left(1\right)}=\tanh \left({\mathbf{z}}_{\sigma }^{\left(1\right)}\right)\approx 0.8957\end{array}$$

Step 2: Bottleneck Distribution Parameters: Compute the mean ($\mu$) and log-variance ($\log {\sigma }^2$), as follows:

$${\mathbf{z}}_{\mu }^{\left(2\right)}={\mathbf{W}}_{\mu }^{\left(2\right)}{\mathbf{a}}_{\mu }^{\left(1\right)}+{\mathbf{b}}_{\mu }^{\left(2\right)}=0.5\times 0.9217+0.2=0.6609\Rightarrow \mu ={\mathbf{z}}_{\mu }^{\left(2\right)}=0.6609$$

$${\mathbf{z}}_{\sigma }^{\left(2\right)}={\mathbf{W}}_{\sigma }^{\left(2\right)}{\mathbf{a}}_{\sigma }^{\left(1\right)}+{\mathbf{b}}_{\sigma }^{\left(2\right)}=0.45\times 0.8957+0.25=0.6531\Rightarrow \log {\sigma }^2={\mathbf{z}}_{\sigma }^{\left(2\right)}=0.6531$$

Step 3: Variance and Standard Deviation: Convert log-variance to variance and standard deviation:

$${\sigma }^2=\exp \left(\log {\sigma }^2\right)=\exp \left(0.6531\right)\approx 1.9215\Rightarrow \sigma =\sqrt{{\sigma }^2}\approx 1.3862$$

Step 4: Latent Code Sampling (Reparameterization Trick): Sample the latent code using the reparameterization trick:

$$\mathbf{z}=\mu +\sigma ·ϵ=0.6609+1.3862\times 0.5=1.3540$$

Key Insight: The reparameterization trick allows us to backpropagate through the random sampling by making the randomness an input ($ϵ$) rather than part of the computation graph.

Step 5: Decoder Hidden Layer Computation: The decoder begins reconstructing from the sampled latent code:

$$\begin{array}{c}\hfill {\mathbf{z}}^{\left(3\right)}={\mathbf{W}}^{\left(3\right)}\mathbf{z}+{\mathbf{b}}^{\left(3\right)}=0.6\times 1.3540+0.3=1.1124,{\mathbf{a}}^{\left(3\right)}=\tanh \left({\mathbf{z}}^{\left(3\right)}\right)\approx 0.8046\end{array}$$

Step 6: Output Layer (Reconstruction): Produce the final reconstruction:

$$\begin{array}{cc}\hfill {\widehat{\mathbf{x}}}_1& ={\mathbf{a}}^{\left(4\right)}={\mathbf{z}}^{\left(4\right)}={\mathbf{W}}^{\left(4\right)}{\mathbf{a}}^{\left(3\right)}+{\mathbf{b}}^{\left(4\right)}=[0.70.8]\times 0.8046+[0.40.5]=\left[\genfrac{}{}{0ex}{}{0.9632}{1.1437}\right]\hfill \end{array}$$

Loss Computation: The VAE has two loss components: reconstruction loss and KL divergence loss. First, the reconstruction loss, as in an AE, measures how well we reconstruct the input:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{recon}}={\displaystyle \frac{1}{2}}{\parallel {\mathbf{x}}_1-{\widehat{\mathbf{x}}}_1\parallel }^2={\displaystyle \frac{1}{2}}\left[{\left(0.0368\right)}^2+{\left(1.8563\right)}^2\right]\approx 1.7236\end{array}$$

and the KL divergence loss measures how much the learned distribution $q\left(\mathbf{z}\right|\mathbf{x})=\mathcal{N}(\mu ,{\sigma }^2)$ deviates from the prior $p\left(\mathbf{z}\right)=\mathcal{N}(\mathbf{0},\mathbf{I})$:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{KL}& ={\displaystyle \frac{1}{2}}\left[{\mu }^2+{\sigma }^2-\log {\sigma }^2-1\right]={\displaystyle \frac{1}{2}}\sum _{k=1}^K\left({\mu }_k^2+{\sigma }_k^2-\log {\sigma }_k^2-1\right)\hfill \\ \hfill & ={\displaystyle \frac{1}{2}}\left[-0.6531+1.9215+{\left(0.6609\right)}^2-1\right]\approx 0.3526\hfill \end{array}$$

Then, the total VAE loss will be the weighted sum of both losses (typically with weight $\beta =1$):

$$L_{\mathrm{total}}=L_{\mathrm{recon}}+L_{KL}=1.7236+0.3526=2.0762$$

Key Insight: The KL divergence term acts as a regularizer, encouraging the learned distribution to be close to the standard normal prior.

For reference, the complete forward transformation—from input to reconstruction—can be expressed compactly as:

$$\begin{array}{cc}\hfill {\mathbf{x}}_1& \stackrel{{\mathbf{W}}_{\mu }^{\left(1\right)}{\mathbf{x}}_1+{\mathbf{b}}_{\mu }^{\left(1\right)}}{\to }{\mathbf{z}}_{\mu }^{\left(1\right)}\stackrel{\tanh }{\to }{\mathbf{a}}_{\mu }^{\left(1\right)}\stackrel{{\mathbf{W}}_{\mu }^{\left(2\right)}{\mathbf{a}}_{\mu }^{\left(1\right)}+{\mathbf{b}}_{\mu }^{\left(2\right)}}{\to }\mu \hfill \\ \hfill {\mathbf{x}}_1& \stackrel{{\mathbf{W}}_{\sigma }^{\left(1\right)}{\mathbf{x}}_1+{\mathbf{b}}_{\sigma }^{\left(1\right)}}{\to }{\mathbf{z}}_{\sigma }^{\left(1\right)}\stackrel{\tanh }{\to }{\mathbf{a}}_{\sigma }^{\left(1\right)}\stackrel{{\mathbf{W}}_{\sigma }^{\left(2\right)}{\mathbf{a}}_{\sigma }^{\left(1\right)}+{\mathbf{b}}_{\sigma }^{\left(2\right)}}{\to }\log {\sigma }^2\stackrel{\exp (·)}{\to }{\sigma }^2\stackrel{\sqrt{·}}{\to }\sigma \hfill \\ \hfill \mu ,\sigma & \stackrel{\mathbf{z}=\mu +\sigma \odot ϵ}{\to }\mathbf{z}\stackrel{{\mathbf{W}}^{\left(3\right)}\mathbf{z}+{\mathbf{b}}^{\left(3\right)}}{\to }{\mathbf{z}}^{\left(3\right)}\stackrel{\tanh }{\to }{\mathbf{a}}^{\left(3\right)}\stackrel{{\mathbf{W}}^{\left(4\right)}{\mathbf{a}}^{\left(3\right)}+{\mathbf{b}}^{\left(4\right)}}{\to }{\mathbf{z}}^{\left(4\right)}={\widehat{\mathbf{x}}}_1\hfill \end{array}$$

5.6.2. Backpropagation: Gradient Computation

Before computing gradients, we highlight how they flow through the VAE:

• Decoder parameters: $({\mathbf{W}}^{\left(4\right)},{\mathbf{b}}^{\left(4\right)},{\mathbf{W}}^{\left(3\right)},{\mathbf{b}}^{\left(3\right)})$: Receive gradients only from ${\mathcal{L}}_{\mathrm{recon}}$, not from ${\mathcal{L}}_{\mathrm{KL}}$. Changing decoder weights does not affect $\mu$ or $\sigma$.
  Therefore, ${\mathcal{L}}_{\mathrm{recon}}$ depends on decoder parameters directly, while ${\mathcal{L}}_{\mathrm{KL}}$ depends only on encoder parameters ($\mu$ and $\sigma$). As a result:

  $${\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{W}}^{\left(4\right)}}}={\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{recon}}}{\partial {\mathbf{W}}^{\left(4\right)}}}+\underset{=0}{\underbrace{{\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{KL}}}{\partial {\mathbf{W}}^{\left(4\right)}}}}}$$
• Encoder parameters (${\mathbf{W}}_{\mu },{\mathbf{b}}_{\mu },{\mathbf{W}}_{\sigma },{\mathbf{b}}_{\sigma }$): Receive gradients from both losses:

  -

  From ${\mathcal{L}}_{\mathrm{recon}}$: Through the sampled $\mathbf{z}$ via the reparameterization trick,

  -

  From ${\mathcal{L}}_{\mathrm{KL}}$: Directly, since KL depends only on $\mu$ and $\sigma$.

• The reparameterization trick enables gradients to flow from the reconstruction loss back to $\mu$ and $\sigma$.

Step 1: Decoder Gradients (Reconstruction Only). Identical to AE:

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{recon}}}{\partial {\mathbf{W}}^{\left(4\right)}}}& =\underset{={\widehat{\mathbf{x}}}_1-{\mathbf{x}}_1}{\underbrace{{\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{recon}}}{\partial {\mathbf{z}}^{\left(4\right)}}}}}{\left({\mathbf{a}}^{\left(3\right)}\right)}^{\top }=\left[\genfrac{}{}{0ex}{}{-0.0368}{-1.8563}\right]\times 0.8046\approx \left[\genfrac{}{}{0ex}{}{-0.0296}{-1.4936}\right]\hfill \\ \hfill {\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{recon}}}{\partial {\mathbf{b}}^{\left(4\right)}}}& ={\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{recon}}}{\partial {\mathbf{z}}^{\left(4\right)}}}=\left[\genfrac{}{}{0ex}{}{-0.0368}{-1.8563}\right]\hfill \end{array}$$

Step 2: Decoder Hidden Layer Gradients (Reconstruction):

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{recon}}}{\partial {\mathbf{z}}^{\left(3\right)}}}& =\left({\left({\mathbf{W}}^{\left(4\right)}\right)}^{\top }{\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{recon}}}{\partial {\mathbf{z}}^{\left(4\right)}}}\right)\odot (1-{\left({\mathbf{a}}^{\left(3\right)}\right)}^2)\hfill \\ \hfill & =[0.70.8]\left[\genfrac{}{}{0ex}{}{-0.0368}{-1.8563}\right]\times (1-{0.8046}^2)\approx -0.5328\hfill \end{array}$$

Step 3: Gradient ThroughSampling ($\mathbf{z}$): The latent code $\mathbf{z}$ has gradients from both reconstruction and KL:

$${\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{recon}}}{\partial \mathbf{z}}}={\left({\mathbf{W}}^{\left(3\right)}\right)}^{\top }{\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{recon}}}{\partial {\mathbf{z}}^{\left(3\right)}}}=0.6\times (-0.5328)\approx -0.3197$$

Gradient for Decoder Hidden Weights ${\mathbf{W}}^{\left(3\right)}$

$${\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{W}}^{\left(3\right)}}}={\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{recon}}}{\partial {\mathbf{z}}^{\left(3\right)}}}·\mathbf{z}=(-0.5328)\times 1.3540=-0.7214$$

Similarly, ${\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{b}}^{\left(3\right)}}}={\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{recon}}}{\partial {\mathbf{z}}^{\left(3\right)}}}=-0.5328$.

Step 4: Gradients for Distribution Parameters ($\mu$ and $\log {\sigma }^2$): Gradients flow to $\mu$ and $\sigma$ from both reconstruction (through $\mathbf{z}$) and KL divergence:

$$\mathbf{z}=\mu +\sigma ·ϵ,\mathrm{so}{\displaystyle \frac{\partial \mathbf{z}}{\partial \mu }}=1$$

and from KL:

$${\displaystyle \frac{\partial {\mathcal{L}}_{KL}}{\partial \mu }}=\mu =0.6609$$

then, the total will be:

$${\displaystyle \frac{\partial L_{\mathrm{total}}}{\partial \mu }}={\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{recon}}}{\partial \mathbf{z}}}·{\displaystyle \frac{\partial \mathbf{z}}{\partial \mu }}+{\displaystyle \frac{\partial {\mathcal{L}}_{KL}}{\partial \mu }}=(-0.3197)·1+0.6609=0.3412$$

The gradient with respect to $\sigma$, from sampling:

$$\mathbf{z}=\mu +\sigma ·ϵ,\mathrm{so}{\displaystyle \frac{\partial \mathbf{z}}{\partial \sigma }}=ϵ=0.5$$

and from KL:

$${\displaystyle \frac{\partial {\mathcal{L}}_{KL}}{\partial \sigma }}=\sigma -{\displaystyle \frac{1}{\sigma }}=1.3862-{\displaystyle \frac{1}{1.3862}}=0.6649$$

then, the total:

$${\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial \sigma }}={\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{recon}}}{\partial \mathbf{z}}}·{\displaystyle \frac{\partial \mathbf{z}}{\partial \sigma }}+{\displaystyle \frac{\partial {\mathcal{L}}_{KL}}{\partial \sigma }}=(-0.3197)·0.5+0.6649=0.5050$$

The gradient with respect to $\log {\sigma }^2$:, we need

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial \sigma }{\partial \log {\sigma }^2}}& ={\displaystyle \frac{\sigma }{2}}={\displaystyle \frac{1.3862}{2}}=0.6931\hfill \\ \hfill {\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial \log {\sigma }^2}}& ={\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial \sigma }}·{\displaystyle \frac{\partial \sigma }{\partial \log {\sigma }^2}}=0.5050\times 0.6931\approx 0.3500\hfill \end{array}$$

Step 5: Encoder Parameter Gradients: for $\mu$ pathway:

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{z}}_{\mu }^{\left(2\right)}}}={\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial \mu }}=0.3412\end{array}$$

then,

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{W}}_{\mu }^{\left(2\right)}}}={{\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{z}}_{\mu }}}}^{\left(2\right)}{\mathbf{a}}_{\mu }^{\left(1\right)}=0.3412\times 0.9217\approx 0.3145,{\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{b}}_{\mu }^{\left(2\right)}}}={\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{z}}_{\mu }^{\left(2\right)}}}=0.3412\end{array}$$

Back-propagation through $\mu$ encoder hidden layer will be:

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{a}}_{\mu }^{\left(1\right)}}}& ={\left({\mathbf{W}}_{\mu }^{\left(2\right)}\right)}^{\top }{\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{z}}_{\mu }^{\left(2\right)}}}=0.5\times 0.3412=0.1706\hfill \\ \hfill {\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{z}}_{\mu }^{\left(1\right)}}}& ={\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{a}}_{\mu }^{\left(1\right)}}}·(1-{\left({\mathbf{a}}_{\mu }^{\left(1\right)}\right)}^2)=0.1706\times (1-{0.9217}^2)\approx 0.0257\hfill \end{array}$$

For $\log {\sigma }^2$ pathway:

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{z}}_{\sigma }^{\left(2\right)}}}& ={\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial \log {\sigma }^2}}=0.3500\hfill \end{array}$$

then,

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{W}}_{\sigma }^{\left(2\right)}}}={\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{z}}_{\sigma }^{\left(2\right)}}}{\mathbf{a}}_{\sigma }^{\left(1\right)}=0.3500\times 0.8957\approx 0.3135,{\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{b}}_{\sigma }^{\left(2\right)}}}={\displaystyle \frac{\partial L_{\mathrm{total}}}{\partial {\mathbf{z}}_{\sigma }^{\left(2\right)}}}=0.3500\end{array}$$

Backpropagation through log variance encoder hidden layer:

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{a}}_{\sigma }^{\left(1\right)}}}& ={\left({\mathbf{W}}_{\sigma }^{\left(2\right)}\right)}^{\top }{\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{z}}_{\sigma }^{\left(2\right)}}}=0.45\times 0.3500=0.1575\hfill \\ \hfill {\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{z}}_{\sigma }^{\left(1\right)}}}& ={\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{a}}_{\sigma }^{\left(1\right)}}}·(1-{\left({\mathbf{a}}_{\sigma }^{\left(1\right)}\right)}^2)=0.1575\times (1-{0.8957}^2)\approx 0.0311\hfill \end{array}$$

Step 6: First Layer Parameter Gradients: The first layer receives gradients from both $\mu$ and $\log {\sigma }^2$ pathways. For ${\mathbf{W}}_{\mu }^{\left(1\right)}$ and ${\mathbf{b}}_{\mu }^{\left(1\right)}$:

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{W}}_{\mu }^{\left(1\right)}}}={\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{z}}_{\mu }^{\left(1\right)}}}{\mathbf{x}}_1^{\top }=0.0257\times \left[\genfrac{}{}{0ex}{}{1.00}{3.00}\right]=\left[\genfrac{}{}{0ex}{}{0.0257}{0.0771}\right],{\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{b}}_{\mu }^{\left(1\right)}}}={\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{z}}_{\mu }^{\left(1\right)}}}=0.0257\end{array}$$

and for ${\mathbf{W}}_{\sigma }^{\left(1\right)}$ and ${\mathbf{b}}_{\sigma }^{\left(1\right)}$:

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{W}}_{\sigma }^{\left(1\right)}}}={\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{z}}_{\sigma }^{\left(1\right)}}}{\mathbf{x}}_1^{\top }=0.0311\times \left[\genfrac{}{}{0ex}{}{1.00}{3.00}\right]=\left[\genfrac{}{}{0ex}{}{0.0311}{0.0933}\right],{\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{b}}_{\sigma }^{\left(1\right)}}}={\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{total}}}{\partial {\mathbf{z}}_{\sigma }^{\left(1\right)}}}=0.0311\end{array}$$

Parameter updates via gradient descent: using learning rate $\eta =0.01$, the decoder updates will be:

$$\begin{array}{cc}\hfill {\mathbf{W}}_{\mathrm{new}}^{\left(4\right)}& =[0.70.8]-0.01\times [-0.0296-1.4936]=[0.70030.8149]\hfill \\ \hfill {\mathbf{b}}_{\mathrm{new}}^{\left(4\right)}& =[0.40.5]-0.01\times [-0.0368-1.8563]=[0.40040.5186]\hfill \\ \hfill {\mathbf{W}}_{\mathrm{new}}^{\left(3\right)}& =0.6-0.01\times (-0.7214)=0.6072\hfill \\ \hfill {\mathbf{b}}_{\mathrm{new}}^{\left(3\right)}& =0.3-0.01\times (-0.5328)=0.3053\hfill \end{array}$$

The encoder parameters (e.g., ${\mathbf{W}}_{\mu }^{\left(1\right)},{\mathbf{b}}_{\mu }^{\left(1\right)},{\mathbf{W}}_{\sigma }^{\left(1\right)},{\mathbf{b}}_{\sigma }^{\left(1\right)}$, etc.) are updated in the same way using the gradients computed in steps 5 and 6.

6. Experimental Results and Discussions

This section presents tests designed to illustrate the practical differences between PCA, AEs, and VAEs. All experiments used the Olivetti Faces dataset [44], chosen for its high-dimensional structure and clear semantic classes. The dataset contains 400 grayscale facial images of 40 distinct individuals (10 images per person). For computational efficiency while preserving essential features, all images were resized to $50\times 50$ pixels, resulting in a 2500-dimensional input space, and standardized to zero mean and unit variance per pixel [3]. These experiments show the shift from linear compression to non-linear reconstruction, and finally to probabilistic generation.

The settings for our models were as follows:

1.

PCA: Using the standard version to reduce the data to K dimensions.

2.

AE and VAE: Both used a deep network with four layers in the encoder and four in the decoder.

• Encoder layers: 2500 → 512 → 256 → 128 → K.
• Decoder layers: $K\to$ 128 → 256 → 512 → 2500.
• We used ReLU and batch normalization to make training faster and more stable.
• Bottleneck dimensions: $K=2$ or 3 in the first experiment, and K was $\{2,5,10,100\}$ dimensions in the second experiment.
• Training: Adam optimizer ($\eta =0.001$, weight decay $={10}^{-5}$), MSE loss, 300–400 epochs, $\beta =1.0$ to control the KL divergence regularization in the VAE.

All models in both experiments were trained on the entire dataset without a train–test split, in order to focus on reconstruction capabilities and latent space structure.

6.1. First Experiment: Latent Space Visualization

This experiment looks at how each model organizes the data in a 2D or 3D “map” (latent space). This helps us to see whether the models group similar faces together.

Quantitative Reconstruction:

Table 1. Reconstruction error (MSE) for 2D and 3D spaces. Lower is better.

Method	Dimensions	MSE
PCA	2	0.5932
PCA	3	0.5476
AE	2	0.3214
AE	3	0.2748
VAE	2	0.4123
VAE	3	0.3587

shows that non-linear models (AE and VAE) reconstruct the faces much better than the linear PCA. The AE has the lowest error because it is purely focused on reconstruction. The VAE has a slightly higher error than the AE because it spends some of its “effort” on organizing the latent space into a smooth Gaussian shape (the KL penalty). As expected, moving from 2D to 3D improves the results for all methods.

Visual Analysis: Figure 9 shows the resulting maps.

• PCA in Figure 9 (first column): Shows a very “flat” or linear structure.
• AE in Figure 9 (second column): Learns much more complex, curved patterns, which help it separate different individuals better than PCA.
• VAE in Figure 9 (third column): Forces the points into a single, circular (Gaussian) cluster. This makes the space “smooth,” which is perfect for generating new faces through interpolation.

6.2. Second Experiment: Data Reconstruction

In this experiment, we tested the models using different latent dimensions: $K\in 2,5,10,100$. This shows how the data can be reconstructed from the compressed latent space with minimal error. The core methodological implementations (PCA, AE, VAE) remained consistent with Experiment 1. The key variation was the evaluation across four latent dimensions: 2, 5, 10, and 100. This design allowed us to observe the following:

• Performance at extreme compression (2D).
• The “sweet spot” for balanced compression (5D–10D).
• Behavior near near-lossless reconstruction (100D).

The experimental workflow comprised three stages: First, data standardization was applied using $X_{\mathrm{scaled}}={\displaystyle \frac{X-\mu }{\sigma }}$. Second, each model was trained with an appropriate methodology: PCA utilized direct eigendecomposition of the covariance matrix, the AE was trained for 300 epochs with the Adam optimizer ($\eta =0.001$), and the VAE was trained for 400 epochs with KL-divergence weighting ($\beta =1.0$). Third, reconstruction performance was evaluated using MSE.

Reconstruction Performance:

Table 2. Reconstruction performance comparison with different dimensions.

Latent Dim	PCA	AE	VAE
2	0.60	0.43	0.34
5	0.45	0.25	0.10
10	0.33	0.20	0.06
100	0.06	0.02	0.04

and Figure 10 show that all methods get better as we give them more dimensions (neurons in the bottleneck). PCA is consistently the weakest because it is restricted to linear transformations. Interestingly, in this specific dataset, the VAE’s regularization actually helped it outperform the AE at lower dimensions (5D and 10D). At 100D, the AE wins slightly, as the VAE’s KL penalty starts to limit its reconstruction perfectness.

While all three methods can reconstruct faces, the VAE achieved the lowest MSE score and, visually, the faces reconstructed using the VAE appear more natural than those generated by the other two methods (PCA and AE). The 10D space seems to be the ′sweet spot′ for this dataset, providing a compression rate of 99.6% compression rate (reducing the original $50\times 50$ = 2500 dimensions to just 10) while still enabling the VAE to reconstruct the faces almost perfectly (MSE = 0.06).

6.3. Discussions

These experiments show a clear path of improvement:

1.

From Linear to Non-linear: PCA is a good baseline, but it is too simple for complex data like faces. The AE uses neural networks to capture curved patterns, greatly improving reconstruction.

2.

From Reconstruction to Generation: While both AEs and VAEs compress and reconstruct data, only the VAE organizes the latent space. By accepting a small “penalty” in reconstruction accuracy, the VAE creates a smooth map that allows us to sample and create brand-new data.

3.

Fundamental Trade-Offs: The experiments illuminate core design tensions: linearity versus expressivity, deterministic versus probabilistic representation, and reconstruction accuracy versus generative capability. The progression from PCA to AE to VAE navigates these tensions, each step introducing new functionality that addresses the limitations of the previous paradigm.

4.

Choosing the Right Tool:

• PCA is best for quick, simple, and interpretable compression.
• AE is best for maximum-fidelity compression and reconstruction.
• VAE is the best choice when you need a structured latent space or want to generate new data.

In summary, the journey from PCA to AE to VAE is a move from simple compression to deep creation. Each step fixes a problem in the previous model, resulting in the powerful generative AI tools that we use today.

7. PCA, AE, and VAE: Similarities and Differences

We have now traveled from the simple math of PCA to the deep learning of AEs and the probability of VAEs. This comparative analysis reveals both the conceptual continuity and the fundamental innovations that define this evolutionary path in representation learning and generative modeling.

7.1. Fundamental Objectives: From Reconstruction to Generation

All three methods try to compress data into a small bottleneck, but they have different reasons for doing so:

• PCA: Linear Compression—It finds the best straight-line directions to represent the data. Its goal is to summarize and visualize data.
• Autoencoder: Non-linear Reconstruction—It uses neural networks to learn flexible, curved patterns. Its main goal is to rebuild the data as accurately as possible.
• VAE: Probabilistic Generation—It models the actual “recipe” (probability distribution) of the data. Its main goal is to create brand-new data that looks like the original.

In short: PCA describes, AEs rebuild, and VAEs generate.

7.2. Mathematical and Architectural Frameworks

• Representation Learning:

  -

  PCA: Uses a linear projection $\mathbf{z}={\mathbf{Z}}_{\mathrm{PCA}}^{\top }\mathbf{x}$, to find the latent $\mathbf{z}$.

  -

  AE: Uses a non-linear encoder $\mathbf{z}=f_{\varphi }\left(\mathbf{x}\right)$ and decoder $\widehat{\mathbf{x}}=g_{\theta }\left(\mathbf{z}\right)$.

  -

  VAE: Uses a probabilistic encoder $q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})=\mathcal{N}(\mathbf{z};{\mu }_{\varphi },{\sigma }_{\varphi }^2)$ that predicts a mean and variance, and a probabilistic decoder $p_{\theta }\left(\mathbf{x}\right|\mathbf{z})$.

• How They Learn:

  -

  PCA: Closed-form solution via eigendecomposition/SVD. No iterative training; no parameters in the neural network sense.

  -

  AE: Gradient-based optimization (backpropagation) of reconstruction loss. Parameters = weights and biases of encoder/decoder networks.

  -

  VAE: Gradient-based optimization of the ELBO. Parameters include both network weights and distribution parameters.

• Loss Functions and Optimization:

  -

  PCA: Minimize $L_2$ reconstruction error.

  -

  AE: Minimize reconstruction loss.

  -

  VAE: Maximize ELBO.

• Latent Space Characteristics: The nature of the latent space fundamentally distinguishes these methods, as illustrated in

  Table 3. Comparison of the latent space (bottleneck).

  Property	PCA	Standard AE	VAE
  Structure	Flat, linear surface	Unstructured, messy	Smooth, organized map
  Interpretability	High (principal components)	Low (black-box)	Moderate (regularized)
  Dimensionality	Fixed by the data rank	Chosen by the user	Chosen by the user
  Rules	Must be perpendicular	None (unconstrained)	Must follow a Gaussian bell curve
  Interpolation	Linear only	Possible, but often jumpy	Smooth and meaningful

  .

7.3. Parameter Complexity and Model Capacity

As we saw in our numerical examples, the models get more complex as we go:

• PCA: Deterministic algorithm—no trainable parameters in the conventional sense. Complexity depends on data matrix operations ($O(min(N{M}^2,N^{2}M))$ for N samples of dimension M).
• AE: Parameter count grows with network architecture. For our example AE with architecture $2\to 1\to 1\to 1\to 2$, 11 parameters. General formula for a fully connected AE:

  $$P_{\mathrm{AE}}=\sum _{l=1}^{L-1}(d_l\times d_{l+1}+d_{l+1})$$

  where $d_l$ is the dimension of layer l.
• VAE: Similar parameter count to an AE of the same architecture, plus distribution parameters. For our example VAE with separate $\mu$ and $\log {\sigma }^2$ pathways, approximately double the encoder parameters of a comparable AE. In practice, VAEs often share most encoder layers and only branch at the end, minimizing this overhead.

Key Insight: While PCA has zero “learnable parameters,” its solution is mathematically optimal for linear tasks. AEs and VAEs trade this optimality for flexibility, learning parameters that enable non-linear transformations at the cost of non-convex optimization.

7.4. Generative Capabilities

The following is the most important difference in how they are used:

• PCA: Not generative. You can rebuild what you have, but you cannot easily create something new and realistic.
• Standard AE: Not inherently generative. Because the latent space is messy and has “gaps,” picking a random point usually results in a blurry or broken image.
• VAE: Explicitly generative. This is what it was built for. By forcing the latent space into a smooth Gaussian shape, the VAE ensures that almost any point you pick will decode into a realistic, new face or image. This turns a “compression tool” into a “generative tool”.

8. Conclusions

In this tutorial, we followed a clear path from Principal Component Analysis (PCA) to Variational Autoencoders (VAEs). We have shown how these models are part of the same family, with each one building on the last.

We began with PCA, a simple and elegant method that works in straight lines. While it is great for summarizing data, its linearity limits its power. Autoencoders (AEs) took the next step by adding neural networks, allowing us to capture complex and curved patterns. However, AEs are mostly meant for rebuilding what they have already seen. They lack the organized structure needed to create brand-new samples. The VAE solved this by adding the language of probability. By using the reparameterization trick and the ELBO loss, VAEs transformed the Autoencoder into a true generative model. The KL divergence penalty acts like a “manager” for the latent space, keeping it smooth and organized so that we can easily create new, realistic data. This progression illustrates a fundamental principle in machine learning: complex modern methods often extend simpler, well-understood foundations. PCA serves as the linear baseline, AEs introduce non-linear flexibility, and VAEs add probabilistic semantics. Each step addresses limitations of the previous approach while maintaining conceptual continuity.

For researchers and students, this path offers a guide on which tool to use:

• Use PCA for simple, fast, and easy-to-understand data summaries.
• Use AEs when you need to compress and rebuild complex data with high accuracy.
• Use VAEs when you want to generate new data or have a very organized latent space.

Looking forward, the journey continues to even more advanced AI models—like diffusion models used in modern image generators. However, the foundational principles that we have covered here remain the same. By building intuition through this step-by-step path, we hope to have made deep generative modeling easy to understand for everyone.