A Modular Perspective on the Evolution of Deep Learning: Paradigm Shifts and Contributions to AI

Abstract

The rapid development of deep learning (DL) has demonstrated its modular contributions to artificial intelligence (AI) techniques, such as large language models (LLMs). DL variants have proliferated across domains such as feature extraction, normalization, lightweight architecture design, and module integration, yielding substantial advancements in these subfields. However, the absence of a unified review framework to contextualize DL’s modular evolutions within AI development complicates efforts to pinpoint future research directions. Existing review papers often focus on narrow technical aspects or lack systemic analysis of modular relationships, leaving gaps in our understanding how these innovations collectively drive AI progress. This work bridges this gap by providing a roadmap for researchers to navigate DL’s modular innovations, with a focus on balancing scalability and sustainability amid evolving AI paradigms. To address this, we systematically analyze extensive literature from databases including Web of Science, Scopus, arXiv, ACM Digital Library, IEEE Xplore, SpringerLink, Elsevier, etc., with the aim of (1) summarizing and updating recent developments in DL algorithms, with performance benchmarks on standard dataset; (2) identifying innovation trends in DL from a modular viewpoint; and (3) evaluating how these modular innovations contribute to broader advances in artificial intelligence, with particular attention to scalability and sustainability amid shifting AI paradigms.

1. Introduction

The development of artificial intelligence (AI) has undergone multiple paradigm shifts and technological breakthroughs, with deep learning emerging as a dominant force through its modular and reusable architectural components. While early AI systems like Newell and Simon’s “Logic Theorist” [1] and knowledge engineering systems such as DENDRAL and MYCIN [2] provided foundational reasoning frameworks, it is deep learning’s inherent modularity that has enabled unprecedented scalability and adaptability. This modularity manifests in reusable architectural blocks (convolutional layers, attention mechanisms), flexible normalization techniques, hybrid algorithm modules, transfer learning capabilities, and optimized training methodologies, making it uniquely suited for contemporary AI systems.

The evolution of deep learning traces back to neuroscience-inspired theories and computational breakthroughs that introduced early modular concepts. Rosenblatt’s Perceptron [3] established single-layer neural networks for linear classification, acting as a foundational building block despite its limitations in nonlinear modeling. Hopfield’s neural network [4] demonstrated how distributed representations could solve complex optimization problems, previewing the modular potential of neural systems. The backpropagation algorithm [5] enabled gradient optimization for multi-layer networks, introducing a modular training mechanism that would later support innovations like residual blocks and attention mechanisms. These early milestones laid the groundwork for modern modular architectures by establishing principles of layered processing and distributed learning. The resurgence of deep learning [6] represents a modern reinterpretation of connectionism, enabled by exponential growth in computational power and big data.

The development of convolutional neural networks (CNNs) exemplifies modularity through standardized convolutional stacks, pooling operations, and residual connections. AlexNet’s groundbreaking victory in the 2012 ImageNet competition [7] demonstrated how modular CNN components could be combined to achieve state-of-the-art performance. Subsequent architectures like VGGNet’s deep 3 × 3 convolutional stacks [8] and ResNet’s residual connections [9] further standardized these modular components, enabling the training of thousand-layer networks. Similarly, the Inception module’s multi-scale convolutions [10,11,12] introduced a new level of architectural flexibility by combining parallel processing pathways—principles that would later influence Transformer-based models through attention mechanisms.

In parallel, sequence modeling evolved from recurrent neural networks (RNNs) to Transformers through modular advancements. While LSTMs [13] and GRUs [14] addressed RNNs’ gradient issues, the Transformer [15] revolutionized NLP with self-attention-based parallel processing, effectively modularizing sequence modeling. Transformers’ attention heads and feed-forward networks function as reusable components that can be scaled across different model sizes (e.g., BERT [16], GPT series [17,18,19]). This modularity has enabled the development of generative AI and multimodal fusion systems by allowing researchers to combine attention mechanisms with CNN-based vision modules or graph neural networks (GNNs).

Modern deep learning progresses through four synergistic branches, each contributing distinct modular innovations: (1) Supervised learning (CNNs, Transformers) leverages standardized convolutional and attention blocks; (2) Generative learning (GANs [20], contrastive learning [21], masked autoencoders [22]) introduces reusable generative heads and contrastive objectives; (3) Reinforcement learning (DQN [23], PPO [24]) develops modular decision-making units like actor-critic architectures; (4) Graph neural networks (GCNs [25], GATs [26]) modularize non-Euclidean data processing through graph kernels and message-passing components. These modular components can be combined across branches, such as integrating GNNs with Transformers in multimodal large language models (LLMs), creating hybrid systems that leverage the strengths of different architectural paradigms.

Despite these advances, deep learning faces persistent challenges that highlight the dual promise and perils of modular design. First, data hunger remains a critical issue, where limited labeled data hinders small-sample applications (e.g., medical imaging [27]). While modularity enables transfer learning to partially address this, the field still struggles with efficient data utilization. Second, the computational intensity of training large models raises sustainability concerns [28]. Modular design can both mitigate and exacerbate this issue; while reusable components reduce redundancy, the proliferation of specialized modules increases overall complexity. Third, interpretability remains elusive, as modular “black-box” systems compound opacity in high-stakes domains [29,30]. Fourth, catastrophic forgetting persists in open-world learning [31], challenging the adaptability of modular systems. Finally, the diversity of deep learning variants has transformed the field into a highly empirical process of modular experimentation and fine-tuning—a phenomenon that highlights both the promise and chaos of modular AI design.

This paper aims to map deep learning’s evolution and its contributions to AI through a modular lens, focusing on how architectural innovations have shaped modern AI systems. By synthesizing insights from WSOP, Google Scholar, and other databases, we seek to empower researchers in navigating the complexity of modern deep learning architectures. Our contributions include (1) summarizing the fundamental modular components of deep learning algorithms and their performance across standard datasets and (2) innovatively categorizing the evolutionary trajectory of deep learning into modular components such as feature extraction methods, normalization techniques, hybrid algorithm modules, transfer learning applications, and optimization strategies. The structure of this paper is as follows: Section 1 provides the introduction; Section 2 reviews deep learning model architectures and their variants from a modular perspective; and Section 3 analyzes innovation directions and how modularity shapes the future of AI development.

2. Literature Review

Existing reviews on deep learning have approached the topic from diverse angles, often emphasizing distinct aspects of the field. To provide a structured overview, we categorize prior work into five sub-themes: foundational reviews, technical challenges, model-specific advances, domain applications, and identified gaps in the existing literature.

2.1. Foundational Reviews

The foundational work by LeCun et al. [6] provided a comprehensive overview of deep learning, focusing on its biological inspiration and capacity to mimic hierarchical feature extraction in the brain. This review highlighted breakthroughs in supervised learning for tasks like image and speech recognition, particularly emphasizing the synergy between neuroscience and artificial intelligence in CNNs and RNNs. These architectures revolutionized pattern recognition and sequential data processing.

Subsequent reviews, including that by Goodfellow et al. [32], further established deep learning within the broader context of machine learning, offering a systematic introduction to neural networks and their applications. Jordan and Mitchell [33] provided a broader perspective on machine learning trends, situating deep learning within the evolving landscape of AI research.

2.2. Technical Challenges

Technical challenges in deep learning have been extensively explored in the literature. Minar [34] and Tian [35] delved into issues such as gradient vanishing/explosion, regularization techniques, and hyperparameter optimization. These reviews often emphasized practical engineering considerations like batch normalization and activation functions to stabilize training and improve generalization. Optimization dynamics have also received attention, with works such as that by Kingma and Ba [36] introducing adaptive optimization algorithms like Adam and that by Sutskever et al. [37] highlighting the importance of initialization and momentum in training deep networks.

2.3. Model-Specific Advances

Model-specific reviews have detailed advancements in various deep learning architectures and techniques. Reviews focusing on attention mechanisms [38], autoencoders [39], RWKV [40], diffusion models [41], and Kolmogorov Arnold Networks [42] have provided in-depth insights. Dosovitskiy et al. [43] demonstrated the application of Transformers to image recognition, while Radford et al. [44] introduced models leveraging natural language supervision for visual tasks. Ho et al. [45] advanced generative models with diffusion probabilistic approaches.

2.4. Domain Applications

Domain-specific applications of deep learning have been highlighted in several reviews. Chen [46] prioritized domain-specific applications, such as medical imaging and autonomous systems, showcasing how deep learning transforms industries through predictive maintenance, personalized healthcare, and real-time decision making. Zhao [47] examined the recent applications of transfer learning and self-supervised learning across various fields, offering guidance for selecting appropriate techniques to address data scarcity.

2.5. Gaps in Existing Work

Despite comprehensive coverage of deep learning in existing reviews, notable gaps remain. First, there is a lack of systematic analysis of ”modular innovations” across the deep learning pipeline. While many surveys detail architectural advancements, they seldom comprehensively analyze innovations in feature extraction, activation function design, or optimization dynamics. For the purpose of this paper, we define a ”module” in deep learning as a self-contained component that performs a specific function within the network, such as input encoding, normalization, activation, attention mechanisms, or output modeling. These modules can be combined and reused across different architectures.

Second, interdisciplinary mechanistic insights—such as the role of dopamine-driven meta-reinforcement learning in both biological and artificial systems [48]—are rarely integrated, leaving gaps in our understanding of how neuroscientific principles inspire algorithmic improvements. Hassabis et al. [49] have explored the intersection of neuroscience and AI, but such interdisciplinary connections are underrepresented in most reviews.

Third, emerging challenges like plasticity loss in continual learning, which are critical for real-world adaptability, are underexplored in broader reviews, despite their profound impact on sample efficiency and catastrophic forgetting. Additionally, interpretability remains a significant concern, with limited exploration of methods for interpreting and understanding deep neural networks [50,51].

This paper addresses these gaps by presenting a unique perspective on deep learning research. We focus on providing a comprehensive analysis of modular innovations and their impact on systemic performance, integrating interdisciplinary insights to offer a more holistic understanding of the field. Through this modular perspective, we aim to highlight how different components of deep learning architectures contribute to the overall performance and adaptability of AI systems.

2.6. Deep Neural Networks and Its Variants

Neural networks, as the core architecture of deep learning, have evolved from basic feed-forward neural networks (FNNs) to complex model architectures that adapt to multimodal data. Traditional FNNs achieve unidirectional mapping from input to output through multilayer perceptrons (MLPs), excelling at simple classification and regression tasks (e.g., house price prediction). However, traditional neural networks lack modeling capabilities for spatial, temporal, or topological structures.

$$y=\sigma ({\mathbf{w}}^{\top }\mathbf{x}+b)$$

(1)

$\mathbf{x}\in {\mathbb{R}}^n$ is the input vector, $\mathbf{w}\in {\mathbb{R}}^n$ is the weighted vector, $b\in \mathbb{R}$ is the bias, $\sigma (·)$ is the activation function (such as Sigmoid, ReLU), and y is the output.

To address this, researchers have proposed a series of neural network variants, such as CNN, RNN, LSTM, GAN, VAE, etc. Residual networks (ResNet) utilize skip connections to resolve gradient vanishing issues in deep networks, establishing themselves as a benchmark in image classification; The capsule network [52] enhances the spatial invariance of entity attribute representations through dynamic routing mechanisms, addressing the limitations of CNNs in handling geometric transformations; Graph neural networks (GNNs) model non-Euclidean data via neighbor aggregation (GCN [25]), attention weighting (GAT [26]), or sampling strategies (GraphSAGE [53]), finding broad applications in social network analysis and drug discovery; Neural Turing Machines (NTMs [54]) simulate the storage–computation separation mechanism of computers using external memory modules, providing a novel paradigm for complex reasoning tasks. The evolution of neural network variants has always been problem-driven: ResNet tackles gradient vanishing, MobileNet optimizes computational efficiency, and GNN breaks the bottleneck in modeling unstructured data.

2.6.1. Autoencoders

Autoencoders (AEs) and their variants are a core paradigm for unsupervised representation learning. The Autoencoder framework contains two major modules: the encoding process and the decoding process. The input sample x is mapped to the feature space z by encoder (g), i.e., the encoding process, and then the abstract feature z is mapped back to the original space to obtain the reconstructed sample x’ by decoder (f), i.e., the decoding process. The optimization objective is then to optimize both the encoder and decoder by minimizing the reconstruction error to learn the abstract feature representation z for the sample input x.

$$f,g={argmin}_{f,g}L(x,f(g(x)))$$

(2)

Basic AEs [5] extract low-dimensional features through latent space mapping, used for dimensionality reduction and denoising. Denoising autoencoders [55] learn robust representations by processing noisy inputs, enhancing data cleaning. Sparse autoencoders [56], with sparsity constraints, improve feature interpretability and are suitable for medical imaging analysis. Recently, autoencoder concepts have been integrated with self-supervised learning. Masked autoencoders (MAEs [22]) reconstruct original pixels from randomly masked image patches, driving vision Transformer pre-training (e.g., ViT-MAE [22]). Contrastive autoencoders (CAEs [57]), combining contrastive learning (e.g., SimCLR [21]), optimize feature space alignment for cross-modal retrieval. These variants, through reconstruction target optimization, probabilistic modeling, and task-driven design, continue to expand autoencoders’ applications in generative models, anomaly detection, and cross-modal alignment.

2.6.2. Convolutional Neural Networks

Convolutional neural networks (CNNs), leveraging their local perception and parameter-sharing properties, have become foundational models in image processing. The following is a simple CNN:

$$O(i,j)=\sum _{m=0}^{M-1}\sum _{n=0}^{N-1}X(i+m,j+n)·K(m,n)$$

(3)

$\mathbf{x}$ is the input matrix (e.g., an image), with dimensions $\mathbf{H}\times \mathbf{W}$. $\mathbf{K}$ is the convolution kernel (filter), with dimensions $\mathbf{M}\times \mathbf{N}$. O is the output feature map, with dimensions $(\mathbf{H}-\mathbf{M}+\mathbf{1})\times (\mathbf{W}-\mathbf{N}+\mathbf{1})$. i, j are the position indices of the output feature map. m, n are the indices of the convolution kernel.

Early CNN architectures such as LeNet and AlexNet validated the effectiveness of stacking convolutional layers, while VGGNet enhanced the granularity of feature representation through stacked 3 × 3 small convolutional kernels. As network depth increased, the gradient vanishing problem became increasingly severe. Residual neural networks (ResNet) introduced cross-layer residual connections, enabling the training of thousand-layer networks. The Inception series improved feature diversity through multi-scale parallel convolutions (e.g., Inception modules). To address practical deployment needs, model lightweighting emerged as a critical research direction. MobileNet [58] reduced computational costs through depthwise separable convolutions, enabling real-time mobile inference. EfficientNet [59] achieved a Pareto-optimal balance via compound scaling of depth, width, and resolution. Current advancements focus on dynamic inference, multimodal fusion, and leveraging Neural Architecture Search (NAS) for automated high-performance design.

For example, the essence of the famous ResNet lies in enabling each network layer to learn residual relationships between inputs and outputs rather than directly learning the target output. The residual unit can be denoted as follows:

$$\begin{array}{cc}\hfill & y_l=h\left(x_l\right)+F\left(x_l,W_l\right)\hfill \\ \hfill & x_{l+1}=f\left(y_l\right),\hfill \end{array}$$

(4)

where $x_l$ and $x_{l+1}$ are the inputs and outputs of the lth residual unit, noting that each residual unit generally contains a multilayer structure. F is the residual function, denoting the learned residuals. $W_l$ is the weight matrix of layer l. $h(x_l)=x_l$ denotes the constant mapping, and f is the ReLU activation function.

2.6.3. Recurrent Neural Networks

As a classical model for sequential data processing, RNNs demonstrate unique advantages in tasks requiring temporal modeling, such as time series analysis, speech recognition, and machine translation, through their hidden state propagation mechanism across time steps. The recurrent architecture dynamically captures temporal dependencies within sequences. However, traditional RNNs struggle to model long-range dependencies due to the gradient vanishing problem. A simple RNN is as follows and consists of an input layer, a hidden layer, and an output layer:

$$\begin{array}{cc}\hfill & S_t=f\left(U·X_t+W·S_{t-1}\right)\hfill \\ \hfill & O_t=g\left(V·S_t\right),\hfill \end{array}$$

(5)

where X is a vector that represents the value of the input layer; S is a vector that represents the output value of the hidden layer; U is the weight matrix from the input layer to the hidden layer; O is also a vector that represents the values of the output layer; V is the weight matrix from the hidden layer to the output layer. The weight matrix W is the weight of the last value of the hidden layer used as the input to this one.

To address these limitations, researchers have developed critical variants: LSTM [60] and GRU leverage gating mechanisms (e.g., input/forget gates) to selectively retain or discard information, significantly enhancing long-term memory capabilities. Bidirectional RNN (Bi-RNN [61]) combines forward and backward sequence context via dual hidden states, improving global semantic understanding. Deep RNN stacks multiple recurrent layers to increase model complexity. Deep RNN, IndRNN [62], and Dilated LSTM [63] extend model complexity and long-sequence modeling capabilities by stacking layers, employing independent neuron recurrence, or using dilation mechanisms, respectively. Additionally, Echo State Networks (ESNs [64]) and Quasi-recurrent neural networks (QRNNs [65]) achieve efficient computation in chaotic system prediction and real-time tasks by randomly initializing reservoirs or leveraging convolutional parallelism. Zhang [66] introduced Recurrent Support Vector Machines (RSVMs), which combine recurrent neural networks (RNNs) with standard Support Vector Machines (SVMs). These improvements focus on three main objectives: alleviating gradient issues, enhancing long-term dependency capture, and optimizing computational efficiency.

Looking ahead, as computational power grows and algorithmic lightweight demands balance, the evolution of RNNs may focus on dynamically adapting to diverse computing environments (e.g., edge devices) and complex data structures (e.g., irregular time series, high-dimensional spaces), sustaining their vitality through the synergy of computational power and algorithms.

2.6.4. Long Short-Term Memory Networks

As the most representative variant of RNNs, the basic LSTM is defined by

$$\begin{array}{cc}\hfill i_t& =\sigma \left(W_{xi}{x}_t+W_{hi}{h}_{t-1}+b_i\right),\hfill \\ \hfill C_t^{*}& =tanh\left(W_{xc}{x}_t+W_{hc}{h}_{t-1}+b_c\right),\hfill \\ \hfill f_t& =\sigma \left(W_{xf}{x}_t+W_{hf}{h}_{t-1}+b_f\right),\hfill \\ \hfill C_t& =f_t{C}_{t-1}+i_t{C}_t^{*},\hfill \\ \hfill o_t& =\sigma \left(W_{xo}{x}_t+W_{ho}{h}_{t-1}+b_o\right),\hfill \\ \hfill h_t& =o_{t}tanh\left(C_t\right).\hfill \end{array}$$

(6)

Compared to basic RNNs, LSTMs introduce the cell state $C_t$, which can be viewed as the current state preserved during the computation of activations $h_t$. Both $c_t$ and $h_t$ are propagated to the next step throughout the process. In the formulas, $i_t,o_t$ and $f_t$ denote the input gate, output gate and forget gate, which are computed using the sigmoid function to produce values between 0 and 1, filtering (retaining or discarding) data. The additional gates in LSTMs are the forget gate $f_t$, input gate $i_t$, and output gate $o_t$, which control the proportion of previous and current cell states and how the cell generates $h_t$.

LSTM has many variants. For example, GRU merges gate units and hidden states to balance efficiency and performance in resource-constrained scenarios. ConvLSTM replaces fully connected layers with convolutional operations to achieve joint spatiotemporal modeling, suitable for tasks like video prediction [67]. xLSTM [68] extends the traditional LSTM by incorporating additional gating mechanisms to better capture long-range dependencies, enhancing performance in sequential data with extended temporal relationships. Vision-LSTM [69] adapts xLSTM building blocks to computer vision. Phased LSTM introduces a time-gating mechanism to adapt to irregularly sampled data [70]. Grid LSTM and MD-LSTM extend traditional LSTMs for multidimensional data, applicable to tasks such as image generation [71,72]. Variants like Tree-LSTM [73] enhance expressiveness in complex tasks by incorporating techniques such as attention mechanisms. These variants reflect a task-driven technical path. For instance, BiLSTM [74] is suitable for NLP tasks requiring global context, while SRU [75], with simplified gates and parallel computation, is ideal for large-scale real-time scenarios.

2.6.5. Graph Neural Networks

Graph neural networks (GNNs) aim to learn representations for each node in a graph. These representations are computed based on the node’s features, edge features, and the features of its neighboring nodes. In graph neural networks (GNNs), the iterative process of updating node features is captured by the equations

$$\begin{array}{cc}\hfill & H^{t+1}=\tilde{A}{H}^{t}W\hfill \\ \hfill & \tilde{A}=I_n+D^{-{\displaystyle \frac{1}{2}}}A{D}^{-{\displaystyle \frac{1}{2}}},\hfill \end{array}$$

(7)

where $H^{t+1}$ represents the node features at the next time step, $\tilde{A}$ is the normalized adjacency matrix that accounts for node degrees to ensure equitable information propagation, $H^t$ is the current feature matrix, W is a learnable weight matrix, $I_n$ is the identity matrix, and D is the degree matrix. This formulation allows for the effective aggregation of neighbor information while preserving node-specific features, which is crucial for learning robust node representations in graph-based tasks. The architecture of GNN is shown in Figure 1.

GNNs can be categorized into several types, including graph convolutional networks (GCNs: spectral-based or spatial-based), graph attention networks (GATs) based on attention mechanisms, graph networks based on gated updates, and graph networks with skip connections. ChebNet [76] was proposed, which uses Chebyshev polynomials to approximate the convolutional kernel, thereby reducing the number of parameters and lowering computational complexity.

2.6.6. Kolmogorov–Arnold Networks

Kolmogorov–Arnold Networks (KAN) are a novel neural network architecture based on Kolmogorov and Arnold’s theorem, which states that any continuous multivariable function can be represented as a superposition of a finite number of continuous univariate functions. Specifically, for an n-dimensional continuous function $f(x_1,x_2,\dots ,x_n)$, there exists a set of continuous univariate functions $g_i$ and $h_{ij}$ such that

$$f(x_1,x_2,\dots ,x_n)=\sum _{i=1}^{2n+1}{g}_i\left(\sum _{j=1}^n{h}_{ij}(x_j)\right)$$

(8)

This theorem indicates that complex multivariate functions can be represented through combinations of simple univariate functions, providing new perspectives for neural network design [77]. Figure 2 illustrates the structure of a KAN.

In recent years, KAN has shown significant progress in fields like time series prediction and medical diagnosis. For instance, the MT-KAN model proposed by [78] uses a two-layer network with spline-parametrized functions to capture complex temporal dependencies in multivariate time series. In medical diagnosis, Tang [79] integrated KAN into the U-Net model (U-KAN) for superior 3D brain tumor segmentation using multimodal MRI data. Shuai [80] introduced the PIKAN model, which replaces traditional MLP layers with KAN layers for more accurate power system dynamics prediction with a smaller network size. Future research will focus on optimizing the KAN’s structure and training methods, exploring new applications in areas like finance and environmental monitoring, and combining it with fields such as quantum computing and bioinformatics.

2.6.7. Bayesian Neural Networks

Bayesian neural networks (BNNs) can be simply understood as introducing uncertainty into the weights of neural networks for regularization. This characteristic makes BNNs excel in handling small datasets, active learning, and combating overfitting.

A neural network can be viewed as a conditional distribution model $P(\mathbf{y}\mid \mathbf{x},\mathbf{w})$: the distribution of the predicted output values $\mathbf{y}$ given the input $\mathbf{x}$, where $\mathbf{w}$ represents the weights within the neural network. Based on the weights $\mathbf{w}$, the probability model for predicting $\mathbf{y}$ from the input $\mathbf{x}$ given dataset $\mathcal{D}$ becomes $P(\widehat{\mathbf{y}}\mid \widehat{\mathbf{x}})={\mathbb{E}}_{P(\mathbf{w}\mid \mathcal{D})}[P(\widehat{\mathbf{y}}\mid \widehat{\mathbf{x}},\mathbf{w})].$ According to Bayesian theory, to determine $\mathbf{w}$ given $\mathcal{D}$, one must proceed through

$$P(\mathbf{w}\mid \mathcal{D})={\displaystyle \frac{P(\mathbf{w},\mathcal{D})}{P(\mathcal{D})}}={\displaystyle \frac{P(\mathcal{D}\mid \mathbf{w})P(\mathbf{w})}{P(\mathcal{D})}}.$$

(9)

Using variational methods, we can approximate the true posterior distribution $P(\mathbf{w}\mid \mathcal{D})$ with a distribution $q(\mathbf{w}\mid \theta )$ controlled by a set of parameters $\theta$. This process can be achieved by minimizing the Kullback–Leibler (KL) divergence between the two distributions.

$$\mathcal{F}(\mathcal{D},\theta )=D_{\mathrm{KL}}[q(\mathbf{w}\mid \theta )\parallel P(\mathbf{w})]-{\mathbb{E}}_{q(\mathbf{w}\mid \theta )}[logP(\mathcal{D}\mid \mathbf{w})].$$

(10)

Current research progress in Bayesian neural networks (BNNs) mainly focuses on improving training efficiency, enhancing uncertainty quantification, and expanding application scenarios. For example, Gal [81] proposed Dropout as a Bayesian approximation method, demonstrating how to estimate model uncertainty through Dropout to enable Bayesian inference in deep learning. Additionally, Blundell [82] introduced a variational inference method for weight uncertainty, further advancing the practical applicability of BNNs. Fortunato [83] proposed Bayesian recurrent neural networks, showing their potential in handling sequential data. Despite this, BNNs have tremendous potential, particularly in scenarios demanding model interpretability and robustness.

2.6.8. Physics-Informed Neural Networks

Physics-informed neural networks (PINNs) are a rapidly growing field that leverages the power of neural networks to learn complex patterns and relationships from data while also incorporating the underlying physical principles such as partial differential equations (PDEs) or ordinary differential equations (ODEs) that govern the system. Let ${\mathcal{L}}_{total}$, ${\mathcal{L}}_{Data}$, ${\mathcal{L}}_{DE}$, ${\mathcal{L}}_{BC}$ and ${\mathcal{L}}_{IC}$ denote total loss, partial differential equation loss, initial condition loss, and boundary condition loss, respectively. Then, as shown in Figure 3,

$${\mathcal{L}}_{total}={\lambda }_1{\mathcal{L}}_{Data}+{\lambda }_2{\mathcal{L}}_{DE}+{\lambda }_3{\mathcal{L}}_{BC}+{\lambda }_4{\mathcal{L}}_{IC},$$

(11)

with parameters ${\lambda }_1, {\lambda }_2, {\lambda }_3,$ and λ₄ representing weights for the adjustment of each loss term.

The extended models of PINN include Spectral PINNs (analyzing vibration phenomena through wavenumber domain techniques to improve analysis accuracy [85]), Hybrid PINNs (combining the computational efficiency of traditional PINNs with data-driven methods to reduce training time and enhance scalability [86]), and Theory-Constrained PINNs (embedding domain-specific theoretical principles to ensure model compliance with physical laws, thereby improving prediction accuracy and explainability [87]). Additionally, PINN has performed excellently in structural analysis, health monitoring, stress–strain analysis, and multi-scale modeling of composite materials, demonstrating high computational efficiency and accuracy [88,89].

2.6.9. Liquid Neural Networks

Liquid neural networks (LNNs) represent a paradigm shift in dynamic information processing, inspired by the adaptability and temporal processing capabilities of biological neurons. The foundational framework can be traced to the concept of Liquid Time-Constant Networks (LTCs [90]), where the system’s state evolves according to

$${\displaystyle \frac{d\mathbf{x}(t)}{dt}}=-\left[{\displaystyle \frac{1}{\tau }}+f(\mathbf{x}(t),\mathbf{I}(t),t,\theta )\right]\mathbf{x}(t)+f(\mathbf{x}(t),\mathbf{I}(t),t,\theta )A,$$

(12)

where $\mathbf{x}(t)$ is the hidden state, $\mathbf{I}(t)$ is the input, t represents time, $\tau$ denotes the time constant, and f is parameterized by $\theta$. A is a parameter matrix. This formulation enables LNNs to adaptively modulate their response latency based on input stimuli, mimicking the behavior of biological neural systems. Applications span robotics, autonomous systems, and real-time signal processing, where handling non-stationary data streams is critical. Figure 4 provides an example of the motivation of LTCs.

To address the computational complexity of solving ODEs, Hasani [91] proposed Closed-form Continuous-time Networks (CfCs). By deriving closed-form approximations of neuronal dynamics, CfCs bypass numerical integration while preserving temporal fidelity. This approach reduces training costs by up to 70% and scales to high-dimensional tasks like video prediction and robotic control. Further studies, such as that by Kumar [92], emphasize LNNs’ ability to model short-term memory and context-dependent computation. By exploiting sparse spikes and adaptive time constants, LNNs achieve sub-millisecond latency in edge computing scenarios, paving the way for real-world embedded systems.

2.7. Deep Generative Models and Variants

2.7.1. Generative Adversarial Networks

Generative adversarial networks (GANs [20]) employ an adversarial mechanism between a generator (G) and a discriminator (D), both implemented as multilayer perceptrons, to synthesize high-fidelity multimodal data. The generator maps noise variables z from a prior distribution $p_z(z)$ to data space via $G(z;{\theta }_g)$, producing a generated distribution $p_g$. Simultaneously, the discriminator $D(x;{\theta }_d)$ outputs a scalar probability $D(x)$ indicating whether x originates from real data or $p_g$. During training, D aims to maximize correct classification of real and generated samples, while G minimizes $log(1-D(G(z)))$, thereby enhancing the generator’s ability to produce realistic data. In summary, D and G engage in a two-player minimax game, where the value function $V(G,D)$ is defined as follows:

$$\underset{\mathrm{Adversarial}\mathrm{game}}{\underbrace{\underset{G}{min}\underset{D}{max}}}V(D,G)=\underset{\mathrm{Real}\mathrm{data}\mathrm{score}}{\underbrace{{\mathbb{E}}_{\mathbf{x}\sim p_{\mathrm{data}}}[logD(\mathbf{x})]}}+\underset{\mathrm{Fake}\mathrm{data}\mathrm{detection}}{\underbrace{{\mathbb{E}}_{\mathbf{z}\sim p_{\mathbf{z}}}[log(1-D(G(\mathbf{z})))]}}$$

(13)

Basic architectures like DCGAN establish an engineering foundation for image generation through convolutional layer normalization. The Wasserstein GAN (WGAN [93]) introduces Wasserstein distance and gradient penalty (GP) to effectively mitigate mode collapse. CycleGAN [94] leverages cyclic consistency loss to perform cross-domain translation with unpaired data (e.g., translating horses into zebras), extending applications to artistic style transfer and medical image cross-modal reconstruction. For generation control and quality improvement, the StyleGAN [95] series enables hierarchical editing of facial details through style mixing and disentangled latent spaces, while ProGAN [96] uses progressive training to gradually increase resolution for high-definition image generation. Conditional GANs (cGANs [97]) guide generation via label embedding, and CLIP-GAN [98] integrates cross-modal pre-trained models (e.g., CLIP) for text-driven image synthesis, advancing controllable generation toward multimodal interaction. Current GAN improvements focus on 3D generation (e.g., 3D-GAN for point cloud generation [99]), training stability optimization (e.g., spectral normalization), and integration with diffusion models (e.g., Projected GAN [100]), continually pushing boundaries in digital content creation, virtual reality, and industrial design.

2.7.2. Boltzmann Machine

The Boltzmann Machine (BM [101]) is a network of symmetrically coupled stochastic binary units. It contains a set of visible units $\mathbf{v}\in {\{0,1\}}^D$ and a set of hidden units $\mathbf{h}\in {\{0,1\}}^P$. The energy of the state $\{\mathbf{v},\mathbf{h}\}$ is defined as

$$E(\mathbf{v},\mathbf{h};\theta )=-{\displaystyle \frac{1}{2}}{\mathbf{v}}^{\top }\mathbf{L}\mathbf{v}-{\displaystyle \frac{1}{2}}{\mathbf{h}}^{\top }\mathbf{J}\mathbf{h}-{\mathbf{v}}^{\top }\mathbf{W}\mathbf{h},$$

(14)

where $\theta =\{\mathbf{W},\mathbf{L},\mathbf{J}\}$ are the model parameters. In this context, $\mathbf{W}$, $\mathbf{L}$, and $\mathbf{J}$ represent visible-to-hidden, visible-to-visible, and hidden-to-hidden symmetric interaction terms. The diagonal elements of $\mathbf{L}$ and $\mathbf{J}$ are set to 0. The probability that the model assigns to a visible vector $\mathbf{v}$ is

$$p(\mathbf{v};\theta )={\displaystyle \frac{p^{*}(\mathbf{v};\theta )}{Z(\theta )}}={\displaystyle \frac{1}{Z(\theta )}}\sum _{h}exp\left(-E(\mathbf{v},\mathbf{h};\theta )\right),$$

(15)

where $p^{*}$ denotes unnormalized probability, and $Z(\theta )={\sum }_{\mathbf{v}}{\sum }_{\mathbf{h}}exp\left(-E(\mathbf{v},\mathbf{h};\theta )\right)$ is the partition function. The conditional distributions over hidden and visible units are given by $p(h_j=1|\mathbf{v},{\mathbf{h}}_{-j})=\sigma \left({\sum }_{i=1}^D{W}_{ij}{v}_i+{\sum }_{m=1\setminus j}^P{J}_{jm}{h}_j\right),$ and $p(v_i=1|\mathbf{h},{\mathbf{v}}_{-i})=\sigma \left({\sum }_{j=1}^P{W}_{ij}{h}_j+{\sum }_{k=1\setminus i}^D{L}_{ik}{v}_j\right),$ where $\sigma (x)=1/(1+exp(-x))$ is the logistic function.

Boltzmann Machines (BMs), inspired by statistical mechanics, use energy functions and simulated annealing for training. However, their fully connected structure results in high computational complexity. Restricted Boltzmann Machines (RBMs [102]) mitigate this issue by restricting intra-layer connections and adopting the Contrastive Divergence (CD) algorithm, enhancing training efficiency. Deep Belief Networks (DBNs [32]), generative models constructed by stacking RBMs, are trained using unsupervised learning to extract hierarchical features, followed by supervised fine-tuning. Tang [103] introduced Deep Lambertian Networks (DLNs), which combine Lambertian reflectance with Gaussian RBMs and DBNs, allowing for the modeling of latent variables like albedo, surface normals, and light sources, thus better simulating the image generation process. The Deep Boltzmann Machine (DBM [104]), which stacks RBMs for complex data modeling, excels in multimodal data modeling and industrial diagnostics through layer-wise pretraining. Recent variants such as the Monotonic Deep Boltzmann Machine (mDBM [105]) integrate monotonicity constraints and parallel inference, achieving high classification accuracy on datasets like MNIST and CIFAR-10. The Born Machine [106], leveraging quantum many-body computation and tensor networks, generates uncorrelated samples via quantum state probabilities. While BM variants are powerful in expressiveness and generative tasks, their applications in quantum computing (e.g., Quantum BMs [107]) and NP-hard problem solving (e.g., for the Traveling Salesman Problem) demonstrate their versatility. However, high training complexity and resource demands remain challenges for future advancements.

2.7.3. Variational Autoencoder

The main principle of the variational autoencoder [108] is to map a set of data into an ideal Gaussian distribution through an encoder. Then, the samples sampled by Gaussian distribution are input into the decoder to generate reconstructed data. The optimization goal of the variational autoencoder is to maximaze the probability $p(x)$ generated by input X by optimizing $\theta$ on the premise of sampling Z. A function $Q(z|x)$ is therefore introduced to perform the function of the coding network.

$$J_{VAE}={\mathbb{E}}_{Q(z|x)}[logP(x|z)]-D[Q(z|x)\Vert P(z)].$$

(16)

Recent advancements in VAE research have focused on improving generative quality, addressing issues like blurry image generation, and enhancing the model’s ability to handle complex data. Improvements include Conditional VAE (CVAE, [109]) for generating specific categories of data, VAE-GAN [110] hybrids for higher-quality image synthesis, and PixelVAE [111] for combining autoregressive decoding with VAEs. Additionally, models like NVAE [112] and CWAE [113] have introduced architectural innovations to reduce training time and improve stability. Future research directions include better information separation, improved interpretability, and reduced computational complexity.

2.7.4. Diffusion Models

Diffusion models consist of a forward process that sequentially corrupts data with Gaussian noise, ultimately transforming the data distribution into pure noise, and a backward process where a denoising neural network is trained to remove the noise and restore a clean data distribution. This framework enables the generation of high-quality samples by learning to reverse the noise addition process. Here we consider the Ornstein–Ulhenbeck process, which is described by the following Stochastic Differential Equation (SDE),

$$\mathrm{d}{X}_t=-{\displaystyle \frac{1}{2}}g(t)X_t\mathrm{d}t+\sqrt{g(t)}\mathrm{d}{W}_t\mathrm{for}g(t)>0,$$

(17)

where the initial $X_0\sim P_{\mathrm{data}}$ follows the data distribution, ${(W_t)}_{t\ge 0}$ is a standard Wiener process, and $g(t)$ is a nondecreasing weighting function. Diffusion models generate fake data by reversing the time of (17), which leads to the following backward SDE:

$$\mathrm{d}{X}_t^{\leftarrow }=\left[{\displaystyle \frac{1}{2}}{X}_t^{\leftarrow }+\nabla log{p}_{T-t}\left(X_t^{\leftarrow }\right)\right]\mathrm{d}t+\mathrm{d}{\overline{W}}_t\mathrm{for}t\in [0,T),$$

(18)

where $\nabla log{p}_t(·)$ is the so-called “score function”, i.e., the gradient of the log probability density function of $P_t$; ${\overline{W}}_t$ is another Wiener process independent of $W_t$; and we use the superscript ← for distinguishing the forward process (17).

Diffusion models have emerged as a powerful generative AI technology, achieving remarkable success in fields such as computer vision, audio generation, reinforcement learning, and computational biology. Recent advancements have primarily focused on conditional diffusion models, which can generate new samples that meet specific conditions. For example, in vision and audio generation, these models enable high-fidelity sample generation with flexible control [45,114]. Additionally, diffusion models are being applied in reinforcement learning to parameterize policies through imitation learning and reward-maximizing planning [115,116]. In the life sciences, conditional diffusion models are used for single-cell image analysis, protein design, and drug discovery, surpassing traditional deep generative models [117,118]. Future research directions include linking diffusion models to stochastic control theory, enhancing adversarial robustness, and exploring distributionally robust optimization and discrete diffusion models.

2.8. Transformer and Its Variants

To address long texts and the inability of RNNs to process information in parallel, the famous attention mechanism was introduced and created the Transformer architecture. The core formula of the attention mechanism is typically expressed as follows:

$$Attention(\mathbf{Q},\mathbf{K},\mathbf{V})=softmax\left({\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^T}{\sqrt{d_k}}}\right)\mathbf{V}$$

(19)

In the Transformer’s attention mechanism, Q (Query), K (Key), and V (Value) are vectors where attention scores are calculated by comparing Q and K, and V is weighted by these scores to produce the output. This allows the model to focus on different input parts when generating output. The division by $\sqrt{d_k}$ scales the scores to prevent softmax domination by large values when the key vector dimension $d_k$ is large.

To address time series complexities, Transformer-based architectures have evolved with specialized mechanisms: Informer [119] employs ProbSparse attention and distillation to reduce redundancy (e.g., electricity forecasting), while AutoFormer [120] leverages autocorrelation for energy demand predictions. For non-stationary data, the Non-stationary Transformer [121] decomposes sequences into stationary/non-stationary components. iTransformer [122] reverses temporal modeling by encoding variables independently, whereas ETSformer [123] integrates exponential smoothing for interpretable decomposition. Medical signal analysis benefits from ShapeFormer’s [124] morphological attention for waveform detection. Beyond temporal domains, lightweight and cross-modal advancements include Deformable DETR’s [125] region-focused attention for vision tasks, Switch Transformer’s [126] trillion-parameter MoE scaling, and Swin Transformer’s [127] shifted-window hierarchical alignment. CrossFormer [128] further unifies image–text granularity. These innovations, summarized in

Table 1. Development of Transformer and its variants.

Model	Improvement	Application Scenarios
Transformer	Self-attention mechanism enabling parallel sequence processing, replacing traditional RNNs	Natural Language Processing (NLP), machine translation, image segmentation
BERT	Bidirectional Transformer with masked language modeling (MLM)	Text classification, question answering
GPT Series	Autoregressive generation with unidirectional Transformer	Text generation, dialogue systems
T5	Unified text-to-text Transformer framework	Multi-task learning, text generation
Longformer	Sparse attention (local sliding window + global attention), reduces complexity to O(n)	Long-text summarization, document understanding
Reformer	Locality-Sensitive Hashing (LSH) for key grouping, reversible residuals for memory efficiency	Genome sequence analysis, music generation
Transformer-XL	Recurrence mechanism (caching previous segments) with relative positional encoding	Language modeling, dialogue systems
Linformer	Low-rank projection to compress key value matrices (O(n) complexity)	Real-time translation, large-scale text processing
Performer	FAVOR+ (Fast Attention Via Orthogonal Random features) for linear time attention	Protein sequence modeling, image generation
BioBERT	Domain-specific pretraining on biomedical/scientific corpora	Medical literature mining, chemical entity recognition
Vision Transformer (ViT)	Image-patching strategy for standard Transformer adaptation	Image classification, object detection
Switch Transformer	Mixture-of-Experts (MoE) with dynamic token routing (trillion-scale parameters)	Large-scale pretraining
Deformable DETR	Deformable attention with dynamic receptive fields for faster convergence	Computer vision (e.g., COCO dataset detection)
Informer	ProbSparse self-attention and distillation for O(L log L) complexity	Long-sequence forecasting (energy consumption, weather)
AutoFormer	Autocorrelation mechanism capturing periodic dependencies	Energy demand forecasting
Non-stationary Transformer	Hybrid attention (stationary/ non-stationary components)	Non-stationary time series prediction
Crossformer	Hierarchical cross-scale attention for multivariate interactions	Traffic flow prediction
iTransformer	Inverted dimension modeling with variable-specific encoding	Multivariate forecasting
ETSformer	Integration of Exponential Smoothing (ETS) decomposition with frequency attention	Medical time series analysis
ShapeFormer	Morphological attention for local waveform patterns	Biosignal classification

, demonstrate Transformers’ adaptability across finance, healthcare, and multimodal systems, driven by co-optimization of algorithmic design and computational efficiency.

2.9. RWKV and Its Variants

The Receptance Weighted Key Value (RWKV) model [129] has been proposed as an efficient alternative to Transformers. RWKV combines the strengths of recurrent neural networks (RNNs) and linear attention mechanisms to achieve comparable performance with significantly reduced computational costs. The RWKV model employs time mixing to capture the relationship between tokens, as illustrated in Figure 5. For time t, given word $x_t$ and the previous word $x_{t-1}$, the Time-Mix module formulas are as follows:

$$\begin{array}{cc}\hfill r_t& =W_r·({\mu }_r\odot x_t+(1-{\mu }_r)\odot x_{t-1}),\hfill \\ \hfill k_t& =W_k·({\mu }_k\odot x_t+(1-{\mu }_k)\odot x_{t-1}),\hfill \\ \hfill v_t& =W_v·({\mu }_v\odot x_t+(1-{\mu }_v)\odot x_{t-1}),\hfill \\ \hfill wk{v}_t& ={\displaystyle \frac{{\sum }_{i=1}^{t-1}exp(-(t-1-i)w+k_i)\odot v_i+exp(u+k_t)\odot v_t}{{\sum }_{i=1}^{t-1}exp(-(t-1-i)w+k_i)+exp(u+k_t)}},\hfill \\ \hfill o_t& =W_o·(\sigma (r_t)\odot wk{v}_t)\hfill \end{array}$$

(20)

The terms $r_t,k_t$ and $v_t$ utilized in this context are analogous to the Q, K, and V components found in Transformer architectures.

RWKV-5 (Eagle) and RWKV-6 (Finch) [130] enhance the model’s expressiveness with multi-head matrix-valued states and dynamic recurrence mechanisms, retaining RNN-like inference efficiency. RWKV7 [131] further improves this with advanced linear attention techniques for lower time and memory complexity in long sequence processing. RWKV has been widely applied across various fields, including medical image restoration (Restore-RWKV [132]), image segmentation (RWKV-SAM [133]), visual language models (VisualRWKV-6 [134]), vision–language representation learning (RWKV-CLIP [135]), and 3D point cloud learning (PointRWKV [136]).

2.10. State Machine and Its Variants

Recent progress in state machines within deep learning focuses on the development and application of Structured State Space Models (SSMs) [137]. Figure 6 is a structural diagram of the SSM. As an efficient sequence modeling method, SSMs have emerged as strong alternatives to RNNs and Transformers, addressing issues like gradient vanishing in RNNs and quadratic complexity in Transformers by processing long sequences with linear or near-linear complexity [137]. Model like Mamba [138], built on the foundational S4 model [139], shows improvements in computational efficiency, memory optimization, and inference speed. SSMs excel in handling long-range dependencies across various fields such as NLP, speech recognition, vision, and time series forecasting. However, challenges in training optimization, hybrid modeling, and interpretability persist.

2.11. Reinforcement Learning Algorithms

Reinforcement learning (RL) involves an agent interacting with its environment to learn optimal behavioral strategies that maximize cumulative rewards. The agent perceives the environment’s state, selects actions, and receives feedback as rewards, aiming to learn a policy through trial and error. State transitions are generally considered random, influenced by the environment, and can be represented by a state transition density function $p(s^{\prime }|s,a)=P(S^{\prime }=s^{\prime }|S=s,A=a)$. The reward, or cumulative future reward U, is defined as $U_t=R_t+R_{t+1}+\cdots$, with the agent’s goal being to maximize this return.

Deep Q-Networks (DQNs [140]) integrated deep neural networks with Q-learning, addressing high-dimensional state spaces. Asynchronous Advantage Actor-Critic (A3C [141]) improves training efficiency through parallel sampling, while Proximal Policy Optimization (PPO [24]) stabilizes policy updates with a clipped objective function. Biologically inspired networks, such as spiking neural networks (SNNs), Deep Belief Networks (DBNs), and Hierarchical Temporal Memory (HTM), draw from neuroscience to suit neuromorphic chips, drive deep feature learning, and excel in temporal pattern prediction. These models, by merging biological mechanisms with computational paradigms, expand AI’s potential in adaptive control, energy efficiency, and brain-like computing.

2.12. Topological Deep Learning and Its Variants

Topological Data Analysis (TDA) applications include integration with machine learning models. For instance, Hajij [142] pioneered the fields of topological machine learning and topological deep learning. Figure 7 depicts the architecture of a three-layer neural network topology. Persistent landscapes and persistent images are key integration points. Unlike conventional methods like principal component analysis and cluster analysis, TDA effectively captures high-dimensional data topology and identifies subtle categories missed by traditional approaches. Recent TDA integration with machine learning focuses on two main aspects. First, topological feature extraction [143] utilizes TDA to extract features from complex datasets, which can then be used for machine learning training. Second, topology-enhanced machine learning algorithms embed topological concepts into existing algorithms to enhance performance. For instance, adding topological features to neural networks has been shown to improve their performance [144,145]. These approaches show great promise in different fields for addressing issues traditionally handled by conventional models.

2.13. Spiking Neural Networks

Spiking neural networks (SNNs) have seen significant advancements in recent years, particularly in terms of scalability, energy efficiency, and hardware integration. Recent research has focused on developing large-scale SNN models, such as Spiking Transformers and Spiking Residual Networks, which have shown competitive performance in tasks like image and speech recognition [147]. These models leverage the unique temporal dynamics of SNNs to process spatiotemporal data more efficiently than traditional artificial neural networks (ANNs) [148,149].

Another key area of progress is the development of neuromorphic hardware and software frameworks that support SNNs. Hardware platforms like Intel’s Loihi and BrainScaleS-2 have enabled the deployment of SNNs on energy-efficient neuromorphic chips, reducing power consumption and improving computational efficiency [150]. Additionally, software frameworks such as SpikingJelly and CogSNN have been developed to facilitate the design and training of SNNs [151].

Training methodologies for SNNs have also seen improvements, including the use of surrogate gradient methods and backpropagation techniques to address the challenges of training deep SNNs [152,153]. These advancements have led to more robust and scalable SNN architectures capable of handling complex tasks with reduced computational overhead [147,154].

In summary, recent advancements in SNNs have focused on enhancing their scalability, energy efficiency, and integration with neuromorphic hardware, making them a promising alternative to traditional deep learning models for a wide range of applications [147,151].

2.14. Benchmarks and Performances of the Deep Learning Models

The Time Series Library (TSLib) contains various benchmarks for different tasks in time series analysis [155]. For the forecasting task, there are long-term and short-term benchmarks. The long-term benchmarks include ETT (with 4 subsets), Electricity, Traffic, Weather, Exchange, and ILI.

In the recent academic literature, the performance of various deep learning models such as CNN, RNN, LSTM, GNN, Transformer, and RWKV has been extensively evaluated on datasets such as ETT, Electricity, Traffic, Weather, Exchange, and ILI. These models are employed for time series forecasting and other sequential data tasks.

In time series forecasting, models like Informer and Autoformer [119,120], which are based on Transformer architectures, have shown significant improvements in handling long-sequence dependencies and have outperformed traditional models in datasets like Electricity and Weather. These models leverage efficient attention mechanisms and decomposition techniques to enhance forecasting accuracy.

LSTM and GRU, which are variants of RNNs, have been widely used in time series forecasting due to their ability to model long-term dependencies [156,157]. However, they often require more computational resources compared to Transformer-based models [158]. Despite this, RNN-based models like LSTM have shown effectiveness in tasks such as weather forecasting and energy consumption prediction. However, their performance in long-sequence tasks is often limited by computational inefficiency. RWKV, a recent model, aims to combine the efficiency of RNNs with the parallelism of Transformers, offering a balance between computational efficiency and performance [129].

Empirical evaluations on datasets like ETT, Electricity, and Weather consistently show that Transformer-based models generally outperform traditional RNN and CNN models in terms of accuracy and efficiency [159,160,161]. However, the choice of model architecture depends on the specific task and dataset characteristics [162].

In summary, while Transformer-based models have demonstrated superior performance in many sequential tasks, the selection of an appropriate model architecture depends on the specific application and data characteristics. For convenience,

Table 2. A glimpse into the variables in Section 2.

Section	Variable in Appearance Order	Explanation
Section 2.6	$\mathbf{x}$	The input vector.
	$\mathbf{w}$	The weighted vector/ matrix.
	b	The bias.
	$\sigma (·)$	The activation function(such as Sigmoid, ReLU).
	y	The output.
	$V(G,D)$	The value function.
	z	The feature representation.
	$f,g$	The decoder and encoder.
	O	The output feature map/ layer.
	F	The residual function in ResNet.
	$h(·)$	The constant mapping.
	S	The output value of the hidden layer in RNN.
	U	The weight matrix from the input layer to the hidden layer.
	V	The weight matrix from the hidden layer to the output layer.
	$C_t$	The cell state.
	$i_t,o_t$, $f_t$	The input gate, output gate, and forget gate.
	$h_t$	The computation of activations in LSTM.
	$H^t$	The node feature matrix in GNN.
	$\tilde{A}$	The normalized adjacency matrix.
	D	The degree matrix.
	$\mathcal{D}$	The dataset.
	$D_{\mathrm{KL}}$	The Kullback–Leibler (KL) divergence.
	${\mathcal{L}}_{total}$, ${\mathcal{L}}_{Data}$, ${\mathcal{L}}_{PDE}$, ${\mathcal{L}}_{BC}$ and ${\mathcal{L}}_{IC}$	The total loss, partial differential equation loss, iniitial condition loss, and boundary condition loss in PINNs.
	${\lambda }_1,{\lambda }_2,{\lambda }_3,{\lambda }_4$	The weights for the adjustment in PINNs.
	$\mathbf{x}(t)$	The hidden state in LNN.
	$\mathbf{I}(t)$	The input in LNN.
Section 2.7	D and G	The discriminator and generator.
	$log(1-D(G(z)))$	The logarithmic loss function.
	$\mathbf{v}\in {\{0,1\}}^D$	The set of visible units in the Boltzmann Machine.
	$\mathbf{h}\in {\{0,1\}}^P$	The set of hidden units in the Boltzmann Machine.
	$\{\mathbf{v},\mathbf{h}\}$	The energy of the state in the Boltzmann Machine.
	$\theta$	The model parameters.
	$\mathbf{W},\mathbf{L},\mathbf{J}$
	$Z(\theta )$	The partition function.
	$\sigma (·)$	The logistic function.
	${(W_t)}_{t\ge 0}$	A standard Wiener process.
	$g(t)$	The weighting function.
	$\nabla log{p}_t(·)$	The score function.
Section 2.8	$Q,K,V$	Query, Key, and Value.
	$d_k$	The dimension of the key vectors.
Section 2.9	$r_t,k_t$ and $v_t$	The components in RWKV.
	$x_t$	The word at t.
Section 2.11	$p(s^{\prime }|s,a)$	The state transition density function.
	s and $s^{\prime }$	The current state and the new state.
	a	The action.
	U	The cumulated future reward.
	$R_t$	The reward at time t.

lists the mathematical notations in this section.

3. Innovation Trends of Deep Learning on AI’s Modular Viewpoint

In Section 2, we systematically reviewed mainstream deep learning models and their variants. Through comprehensive analysis of extensive literature, we posit that current innovations in deep learning models primarily manifest in the following aspects. The landscape is illustrated in Figure 8.

3.1. Innovations in Feature Extraction Methods

The technical evolution in this field mainly concerns adjustment of convolution kernel and attention mechanisms. For CNNs, foundational advancements include nonlinear activation enhancements (AlexNet [7]), small kernel stacking (VGG [8]), multi-scale architectures (Inception [10]), and depthwise separable convolutions (Xception [163]). Subsequent innovations expanded residual modules (ResNeXt [164]) and optimized efficiency via grouped convolutions (ShuffleNet [165]). Feature fusion mechanisms integrated wavelet transforms [166] for frequency domain analysis, while dynamic convolution [167] enhanced expressiveness through parallel kernel weighting. Recent strides in multimodal fusion and geometric adaptation include the Dilated Convolutional Transformer [168] for the global context, Deformable Convolution [169] for geometric robustness, and CVOCA [170] for joint amplitude-phase modeling, collectively advancing CNN performance in parameter efficiency, computational complexity, and feature representation.

Innovations in the attention mechanism evolve through multi-scale fusion, spatiotemporal–frequency integration, and cross-modal alignment. Attention mechanisms include the spatial, such as CBAM, STN; channels, such as SE-Net, ECA-net; the temporal, such as BTA, CTA, HTA; and frequency, such as FFT. We can also divide them into self-attention, cross-attention and multi-head attention according to their mechanism. Early milestones encompassed saliency maps [171], RNN-based attention [172] and alignment–translation synergy [173]. Abundant variants emerged afterwards. Recent work includes high-frequency noise reduction via DCT [174], channel interaction optimization (ECANet [175]), and rotational positional embeddings (RoPE [176]). Cross-modal frameworks like Flamingo [177] enable image–text alignment, while Ring Attention [178] scales to million-length contexts. Multi-scale advances feature SENet [179] and CBAM for channel–spatial recalibration, alongside frequency–ramp architectures [180] balancing high/low-frequency details. The Swift Parameter-free Attention Network (SPAN [181]) is another recent innovation, which dynamically generates attention weights in a parameter-free manner while maintaining efficient computational performance. These innovations address computational efficiency, long-range dependencies, and multimodal alignment, solidifying attention mechanisms as cornerstones of modern deep learning.

3.2. Innovations in Normalization Methods

Early research on normalization methods aimed to calibrate distributions across feature dimensions. Batch Normalization (BN) [182] addressed internal covariate shift using mini-batch statistics, enhancing convolutional network training. Layer Normalization (LN) [183] applied normalization across feature dimensions, stabilizing training in sequential models and Transformers. Instance Normalization (IN) [184] independently normalized channel features, improving visual quality in style transfer. To mitigate BN’s limitations in small-batch scenarios, Group Normalization (GN) [185] grouped channels, while Weight Normalization (WN) [186] decoupled vector parameters to optimize gradient flow. Cross-Iteration Batch Normalization (CBN) [187] further aggregated statistics over multiple iterations. RMSNorm [188] simplified the normalization process by using the root mean square to normalize the input. DeepNorm [189] stabilized training in deep Transformers through residual connection standardization. Dynamic Tanh (DyT) [190] introduced a learnable scaling parameter leveraging the bounded nature of the tanh function. Filter Response Normalization (FRN) [191] enabled online learning without batch statistics via TLU activation integration, while Virtual Batch Normalization (VBN [192]) established paradigms for streaming data through dynamic population statistic estimation.

The evolution of dynamic adaptation strategies also emerged. The FiLM module [193] pioneered input-conditioned parameter generation. Jing [194] expanded multimodal control in generative tasks through dynamic instance normalization (DIN) with external condition vectors. Luo [195] achieved autonomous BN/LN/IN selection via differentiable gating mechanisms in Switchable Norm. Karras [196] enhanced generative diversity in StyleGAN through projected instance normalization (PIN) with style vector projections. Park’s [197] spatially-adaptive normalization (SPADE) became a milestone in image synthesis through semantic map-guided feature modulation. Miyato [198] revolutionized GAN stability with spectral normalization (SN) via weight matrix spectral norm constraints.

Research on normalization methods currently demonstrates three directions: (1) Decoupling from batch dependencies through statistical disentanglement strategies; (2) Developing dynamic parameter modulation based on input features/external conditions; (3) Constructing distributed normalization architectures for multimodal collaboration.

3.3. Innovations in Algorithm Module Enhancement and Hybridization

3.3.1. Architecture Modification

SPPNet [199] introduced a multi-scale pooling layer, which allows the network to process input images of arbitrary sizes and enhances the model’s robustness. ShuffleNet [165] adopted group convolution to reduce computational load and combined channel shuffle operations to enhance feature interaction. ResNet [9] first proposed the residual network module, where residual skip connections solved the training stability issue of extremely deep networks through identity mapping. Its variants, such as ResNeXt [164], expanded the cardinality through group convolution to balance network width and depth. Stochastic Depth [200] further introduced a random layer dropout strategy during training to enhance the generalization ability of ultra-deep networks. Subsequent research presents task adaptation trends. The Multi-Kernel Inverted Residual module (MKIR [201]) achieved efficient local feature fusion through multi-kernel depthwise convolution and attention mechanisms. DenseNet [202] extended the residual idea through dense cross-layer connections to promote feature reuse. Additionally, Swin Transformer [127] achieved multi-scale modeling through hierarchical window attention. Deep Layer Aggregation (DLA [203]) enhanced feature transfer efficiency through cross-layer iterative fusion. RevNet [204] significantly reduced memory usage through reversible residual blocks to support large-scale generative tasks.

3.3.2. Automated and Lightweighting Design

Neural Architecture Search (NAS) methods such as SMASH [205] dynamically generated candidate structures through a super network. EfficientNet [59] proposed a compound scaling rule to optimize depth, width, and resolution uniformly with NAS, promoting model adaptation in resource-sensitive scenarios. Dynamic Routing Networks [206] achieved dynamic trade-offs between computational efficiency and accuracy through input-adaptive path selection mechanisms. MixConv [207] integrated multi-scale depthwise separable convolution kernels and collaborated with the bottleneck compression strategy of DenseNet-BC to balance receptive field diversity and computational cost in mobile detection tasks. RepVGG [208] used multi-branch topological structures to enhance model performance and converted them to single-path architectures through parameter fusion during inference. Lightweighting is mainly about network compression. This field encompasses multiple families of methods, such as quantization [209], decomposition [210], knowledge distillation [211], and network pruning [212]. Knowledge distillation has become a crucial technique in the realm of large language models (LLMs). Distillation techniques have evolved to encompass a wide range of approaches including response distillation such as Decoupled Knowledge Distillation [213]), feature distillation such as FitNets [214], Attention Transfer [215], Overhaul Distillation [216]), relational distillation like RKD [217], Contrastive Distillation [218]), adversarial distillation [219], self-distillation including BYOT, Deep Mutual Learning [220]), and dynamic distillation such as Dynamic Knowledge Distillation [221] and Online Knowledge Distillation [222].

3.3.3. Module Integration and Stitching

The main methods for model stitching include serial connection, parallel connection, interactive combination, and multi-scale fusion. There exists a partial overlap between model stitching and innovations on architectural modifications. Specifically, serial connections such as the stacking of residual blocks in ResNet [9] and cross-layer feature concatenation in DenseNet [202] solve the network depth bottleneck through gradient optimization and feature reuse. Parallel connections such as the multi-branch Inception modules in GoogLeNet [10] and cardinality expansion in ResNeXt [164] enhance model expressiveness through group convolution and multi-path feature aggregation. Interactive combinations such as the parameter sharing between bidirectional self-attention layers in BERT [16] and the skip connections and encoder–decoder feature fusion in U-Net [223] improve module collaboration efficiency through dynamic information crossover. DetectoRS [224] proposed a recursive feature pyramid (Recursive-FPN) that iteratively optimizes multi-scale features through loop connections. Additionally, multi-scale fusion technology has evolved from the unidirectional pyramid of FPN [225] to the bidirectional cross-layer connections of PANet [226] and then to the auto-search architecture of NAS-FPN [227]. Combined with the high-resolution retention strategy of HRNet [228], it forms a cross-scale modeling paradigm that balances spatial and semantic information. These stitching methods have shifted from static stacking to dynamic coupling, providing a flexible and robust feature integration framework for complex tasks. REALM [229], i.e., end-to-end retrieval-enhanced pretraining, enhances the model’s knowledge acquisition capability through joint optimization of the retriever and generator. RETRO [230] with block retrieval and cross-attention mechanisms dynamically integrates external knowledge bases into the generation process. The combination of GANs (generative adversarial networks) and LSTMs (Long Short-Term Memory networks) fully leverages the GAN’s strength in generating complex data distributions and LSTM’s ability to handle long-term temporal dependencies. In recent years, research combining different DL algorithms has made significant progress. For example, the Transformer-enhanced Kalman Filter [231] estimates the hidden states of the Kalman Filter using a Transformer and performs well in time series prediction and anomaly detection. The latest KalmanFormer [232] directly learns the Kalman gain through the Transformer architecture, significantly improving performance in multi-sensor information fusion, achieving high-precision state estimation even in cases of model mismatch and nonlinearity. Cao [233] showed that Transformers can autoregressively generate sequences in Bayesian networks through maximum likelihood estimation (MLE), indicating their significant potential in learning complex probabilistic models and sequence generation tasks. Han [234] proposed a novel method of combining TDA with convolutional neural networks (CNNs) to enhance feature extraction and classification performance.

3.4. Innovations in Optimization Methods

Innovations in deep learning optimization methods primarily encompass parameter optimization, optimizer advancements, and activation function design.

3.4.1. Parameter Optimization

Among gradient-based optimization methods, Stochastic Gradient Descent (SGD) and Nesterov Accelerated Gradient (NAG) [235] are two prominent approaches. Adaptive optimization algorithms bridge momentum methods with learning rate adaptation. Notable examples include Adam [36], which has inspired variants like AdamW [236] and AdaDelta [237]. RMSProp [238], by integrating momentum and dynamic learning rate adaptation, has become a cornerstone for training complex tasks.

The field of optimization has witnessed the emergence of various advanced techniques, which can be categorized into several groups: Second-order optimization methods, such as Newton’s method, Kronecker-factored approximate curvature [239], and L-BFGS [240], have been developed to leverage second-order information for faster convergence. Hybrid methods, including Rectified Adam [241], Evolved Sign Momentum [242], and Second-order Clipped Stochastic Optimization [243], combine different optimization strategies to enhance performance. Population-based methods, such as Evolutionary Strategies [244], are heuristic algorithms [245,246] inspired by biological evolution. They optimize model parameters and offer unique advantages in complex optimization scenarios. Learning rate schedulers, such as Cyclical Learning Rates [247] and One-Cycle Policy [248], have also gained attention for their ability to accelerate training and improve model performance. Recent advancements include strategies like dynamic routing, proxy (sorragate) paths, parameter reparameterization (e.g., RepVGG, [208]), and Neural Architecture Search [59,205], achieving a balance between training efficiency and inference speed.

3.4.2. Activation Function Design

Activation functions have evolved from simple nonlinear mappings to adaptive probabilistic formulations. Early research includes sigmoid, Hyperbolic Tangent (Tanh), and softmax. In the 2010s, non-saturating activation functions and adaptive activation functions emerged, including ReLU [249], Leaky ReLU [250], Exponential Linear Unit (ELU [251]), Swish [252], and GELU [253]. ReLU [249] mitigated the vanishing gradient problem through sparse activation. GELU [253] incorporated stochastic regularization via smooth gating mechanisms, such as Gaussian error functions, aligning with Transformer self-attention computations. Swish [252] enhanced adaptability by dynamically adjusting activation states through learnable parameters. In the 2020s, dynamic and conditional activation functions flourished, such as SwiGLU [254], Mish [255], Dynamic ReLU [167], VeLU [256], and FReLU [257]. SwiGLU [254], combining Swish and Gated Linear Units (GLUs), improved model performance in Transformer-based architectures. VeLU (Variance-enhanced Learning Unit) [256] proved able to dynamically scale based on input variance by integrating ArcTan-Sin transformations and Wasserstein-2 regularization. Current research explores innovative activation functions like sine-based, learned, and rational activations. See

Table 3. Activation functions at a glance: formulations, properties, and usages.

Function	Equation (Forward)	Range	VD	Cost	Best Practice/Pairing
Classic (‘80-’00)
Sigmoid	$\sigma (x)={\displaystyle \frac{1}{1+e^{-x}}}$	(0,1)	H	low	Prob. output, shallow nets
Tanh	$tanh(x)$	(−1,1)	H	low	RNN pre-2010
Softmax	${\displaystyle \frac{e^{x_i}}{{\sum }_j{e}^{x_j}}}$	(0,1)	–	med	Final layer, multi-class
Non-saturating/adaptive (2010s)
ReLU	$max(0,x)$	[0,∞)	L	very low	Default CNN/FC
L-ReLU	$max(\alpha x,x),\alpha =0.01$	(−∞,∞)	L	very low	Sparse/audio nets
ELU	$x(x\ge 0),\alpha (e^x-1)(x<0)$	(−$\alpha$,∞)	L	low	Smooth zero-centre
Swish	$x·\sigma (\beta x),\beta =1$ or learn	(−∞,∞)	L	low	Deep CNN, NAS-found
GELU	$x\Phi (x)=x·{\displaystyle \frac{1}{2}}[1+\mathrm{erf}(x/\sqrt{2})]$	(−∞,∞)	L	med	Transformers, BERT, GPT
Dynamic/conditional (2020s)
SwiGLU	$\mathrm{Swish}(xW)\odot (xV)$	(−∞,∞)	L	med–high	FFN inside Transformers
D-ReLU	$max({\alpha }_{k}x+{\beta }_k,x)$, k = input cond.	[0,∞)	L	med	Mobile CNN, few-shot
VeLU	$\mathrm{ArcTan}(\mathrm{Sin}(x))·\gamma (\mathrm{Var}[x])$	(−$\pi /2$,$\pi /2$)	L	med	Variance-sensitive tasks
FReLU	$max(x,\mathcal{T}(x))$, $\mathcal{T}$ = spatial context	[0,∞)	L	low	Object detection, Seg
Mish	$x·tanh(ln(1+e^x))$	(−∞,∞)	L	low	General-purpose CNN

Vanishing derivative risk: H = high, M = moderate, L = low. For Softmax the derivative is bounded away from zero in its domain, so the entry “–” indicates “not applicable/no VD issue”.

for a detailed comparison.

3.4.3. Techniques to Prevent Overfitting

To prevent overfitting, various techniques are employed in the field of deep learning. Regularization methods, such as L1/L2 regularization [258], Elastic Net [259], gradient penalty [260], and spectral norm regularization [198], are commonly used. Early stopping strategies, including dynamic patience [261], checkpoint ensemble [262], stochastic weight averaging (SWA [263]), and bootstrap validation [264], also play a significant role in preventing overfitting. Another prominent technique is Dropout [265] and its variants, such as Spatial Dropout [266], Drop-connect [267], Zoneout [268], Alpha Dropout [269], Stochastic Depth [200], Variational Dropout [270], Gaussian Dropout [271], Monte Carlo Dropout [81], Concrete Dropout [272], etc., randomly deactivate neurons. Different dropout methods can be applied depending on whether it is a CNN, RNN or any other neural network model. For example, Maxout activation functions [273] accelerate training and improved model performance. Zoneout introduced regularization for RNNs. Recent advanced techniques include data-enhancement techniques such as AutoAugment [274] and Mixup [275] or Cutmix [276]. Some architecture innovations are also helpful. For example, Deep Residual Learning (ResNets [9]) reduced training errors through residual connections. Batch Normalization [182] and its extension, which has been introduced, mitigated internal covariate shifts and accelerated convergence. Neural Architecture Search (NAS) can improve the generalization ability of the model. As stated in Section 3.3.2, distillation techniques can improve the robustness of the model.

3.4.4. Efficient Adaptation and Modularization

Mixture-of-Experts (MoE [277]) systems leveraged conditional computation for sparse activation, reflecting a shift from single-modality to multi-task, multi-scenario generalization. Recent developments include Sparse MoE [278], Gshard [279], BASE [280], Multi-gate MoE [281], Multimodal MoE [282], Mixture of Attention Heads (MOA [283]) and Federated MoE [284]. LoRA (Low-Rank Adaptation, [285]) enabled efficient parameter fine-tuning by freezing backbone networks and training only low-rank adapters, becoming a standard for lightweight fine-tuning. Adapter [286] inserted lightweight modules into frozen pretrained models to support multi-task adaptation. EfficientNet [59] introduced compound scaling rules to balance depth, width, and resolution for model efficiency. FlashAttention [287] optimized GPU memory usage, accelerating attention computations and reducing memory overhead, and it is widely adopted in GPT-4 and LLaMA training. MobileNetV2 [288] employed inverted residual structures for efficient computation on mobile devices.

3.5. Innovations in Transfer Learning Applications

The advancement of deep learning in transfer learning is manifested through the deep integration of theoretical frameworks and engineering practices; foundational mathematical principles were established through early studies on Maximum Mean Discrepancy (MMD, [289]) and adversarial domain adaptation (e.g., Wasserstein distance optimization [93]). Meanwhile, pre-trained models such as BERT [16] and Vision Transformer [43] revolutionized the “pre-training–fine-tuning” paradigm, enabling efficient cross-task transfer via parameter reuse. In multimodal transfer and cross-domain adaptation, CLIP [44] aligns cross-modal feature spaces through contrastive image–text learning, endowing models with zero-shot transfer capabilities. For few-shot and unsupervised transfer scenarios, meta-learning (MAML [290]) and self-supervised techniques (SimCLR [21]) enhance performance under data scarcity via task generalization and augmented signal generation, respectively. In dynamic and incremental transfer, methods like Elastic Weight Consolidation (EWC [291]) mitigate catastrophic forgetting in continual learning, facilitating progressive knowledge accumulation in open environments.

Recently, we have witnessed transfer learning (TL) demonstrating remarkable advantages in addressing data scarcity and domain discrepancies through cross-domain knowledge transfer. Breakthroughs in medical applications stand out. Med3D [292], as shown in Figure 9, is a pre-trained 3D convolutional network trained on the diverse 3DSeg-8 medical imaging dataset; it accelerates and enhances performance in 3D medical tasks like lung/liver segmentation. Med3D outperforms models trained from scratch or on non-medical datasets (e.g., Kinetics), reducing training time by ten times and improving accuracy by 3–20%. Bargshady [293] achieved 94.2 percent accuracy in COVID-19 detection using an Inception-CycleGAN-based model. Kathamuthu [294] elevated detection accuracy to 98 percent via a VGG-16 architecture. Industrial inspection innovations focus on adversarial learning and multimodal fusion, exemplified by Michau [295] achieving a state-of-the-art 99.8 percent performance in anomaly detection through adversarial TL frameworks, and Zhang [296] realizing 99.8 percent accuracy in equipment fault diagnosis via a blockchain-integrated federated learning framework. Satellite remote sensing has overcome data distribution disparities through domain adaptation, with Chen [297] reporting 90.17 percent mean Average Precision (AP) in aircraft detection using a domain-adaptive R-CNN. Environmental science advances feature cross-modal transfer, as evidenced by Cao [298] achieving 99.3 percent accuracy in waste classification using an improved Inception-V3 model. As illustrated in Figure 10, Cao [299] introduced transfer risk as a metric to assess transferability in cross-domain applications. Applied to financial tasks (e.g., stock prediction, portfolio optimization), transfer risk efficiently identifies optimal source tasks (e.g., cross-continent/sector) and correlates strongly with model performance, enabling effective transfer learning in finance. Negative transfer [300], where a model’s performance on downstream tasks declines due to source–target data mismatches, is a problem in transfer learning. In time series contexts, this phenomenon has not been delved into enough.

4. Conclusions

By summarizing paradigms of and advances in DL structures, this paper systematically summarizes and updates the recent fundamentals of deep learning algorithms and compares their performance on standard datasets. We also identify trends in innovation in deep learning algorithms from a modular AI perspective and analyze how these innovations contribute to the advancement of the field.

Through our analysis, we believe deep learning (DL) will continue to significantly impact AI’s modular development. We observe that future innovations in DL will be problem-driven due to the rapidly increasing demand for AI. Innovations will continuously emerge in areas such as feature extraction methods, normalization methods, module combination or stitching, optimization, and transfer learning applications. However, these innovations will not be isolated as before but will be viewed from the perspective of module combination. The siloed innovation of DL models must evolve into a modular collaborative framework to drive next-generation technological advancements. Furthermore, given limited data samples, real-time generated data, and improved hardware situations, the following specific promising directions of DL are also notable.

Few-shot learning and meta-learning are expected to remain critical. These methods enable models to quickly adapt to new tasks using minimal labeled data, thus improving both learning efficiency and generalization. Notable approaches such as Meta-Learner LSTM and MAML have already shown strong results in few-shot learning settings. Further progress will depend on refining model architectures and training strategies under data-scarce conditions. Self-supervised learning is becoming increasingly vital for utilizing large-scale unlabeled datasets. Improving the ability of pretrained models to exploit this type of data can significantly enhance performance across a wide range of AI tasks. Automation and model lightweighting are also key directions. Techniques such as Neural Architecture Search (NAS) aim to automate the design of neural networks, reducing human effort and improving efficiency. Moreover, developing models that can process multiple modalities cohesively will be essential for improving semantic understanding through enhanced cross-modal fusion. Integration of reinforcement learning (RL) with deep learning is crucial for creating intelligent agents capable of making decisions in complex and dynamic environments. This synergy will continue to advance the capabilities of AI systems in real-world scenarios.