Advances in Deep Learning-Driven Metasurface Design and Application in Holographic Imaging

Abstract

Currently, the integration of deep learning technology with metasurface holographic imaging technology has propelled the development of optical imaging. Owing to the precise control of metasurfaces over the characteristics of light waves, holographic imaging technology can produce corresponding three-dimensional images after processing. Therefore, their integration enables the acquisition of high-quality images. The number of articles on metasurface design using neural network-based deep learning methods is increasing day by day; however, reviews on this topic remain scarce. This review introduces the development of neural networks and the relevant content on metasurface design using the four types of networks and the applications of deep learning-designed metasurface holographic imaging technology, thereby enhancing readers’ systematic understanding of such technologies.

1. Introduction

Early studies on metamaterial fabrication primarily focused on controlling the three-dimensional parameters of structures [1,2,3,4,5,6,7,8,9,10,11,12,13]. However, due to high losses and complex fabrication processes during preparation, the concept of metasurfaces gradually emerged through the planarization of metamaterials [14,15,16,17,18,19,20,21,22,23,24,25,26,27]. As metasurfaces exhibit novel physical phenomena [28,29,30,31,32,33,34,35,36,37,38], thus, various novel applications have emerged, such as metagratings developed for optical wavefront shaping and beam splitting [39,40,41,42,43]. Traditional metasurface design often relies on simulation software for structural modeling, where Maxwell’s equations are solved to obtain results. Most methods depend on numerical simulations such as finite element method (FEM) analysis [44] or finite difference time domain (FDTD) [45], which involve lengthy operation cycles and high professional requirements for designers. Structural design and modeling thus remain challenging tasks. Classical approaches primarily seek optimal solutions within vast design spaces and achieve target optical responses through computationally intensive iterative numerical simulations. In recent years, programmable metasurfaces [17,18,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60]—a technology that allows dynamic control of metasurfaces—has emerged. Because of the progress of innovations like deep learning (DL) [61,62], as well as advancements in photonics-related technologies [63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84] and insights from deep neural networks (DNNs) that harness billions of tunable parameters, this development establishes input–output mappings through large-scale, comprehensive training sets and enables high-precision predictions of new data via parametric optimization—ultimately leading to the fundamental transformation of traditional simulation-based design methodologies [85,86,87,88,89,90,91,92]. DNNs exhibit exceptional parameter adjustability, plasticity, diversity, and generalizability, allowing them to effectively generalize using their inherent nonlinear mapping capabilities when confronting unseen data or diverse scenarios [93,94,95,96,97,98]. Although DNNs have previously achieved favorable results in establishing correlations between nanostructures and electromagnetic behaviors, these correlations are often asymmetric and lack absolute determinism due to material parameter influences. Advanced computational models now enable researchers to more accurately predict metasurface optical properties, design superior structures and schemes, and even uncover novel structural configurations beyond those traditionally known.

Holographic imaging is a technique that records the optical wave information of objects using interference principles, enabling the generation of holograms and the reconstruction of target images. Early holographic imaging primarily relied on optical components to modulate light waves. However, with advancements in science and technology, demands for optical systems have increased, driving the trends of miniaturization and integration [99,100,101,102,103,104,105]. Traditional optical components modify light waves via optical path lengths, but they sometimes fail to meet requirements, spurring the development of metasurface technology. The integration of deep learning with metasurface holographic imaging enables rapid metasurface design through deep learning methods, thereby achieving precise control over the amplitude [106], phase [107], and polarization [108] of light fields. This facilitates diverse optical functions such as focusing, beam shaping, and aberration compensation. Metasurface holographic imaging technology finds applications in optical communication, imaging, sensing, display, and other fields [109,110,111,112,113,114], further enhancing the functionality and performance of optical systems. At present, the incorporation of deep learning, along with its rapid development, has indirectly driven corresponding advancements in metasurface holographic imaging technology, leading to three major categories of holograms: pure phase holograms [115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135], pure amplitude holograms [136,137,138,139], and complex amplitude holograms [140,141].

The integration of deep learning with metasurface design marks a convergence between metasurfaces and cutting-edge artificial intelligence, driving innovative development in metasurfaces. This review analyzes the latest progress in deep learning for metasurface modeling, contrast calculation, and design, providing reference solutions for metasurface advancement while summarizing relevant research achievements. Additionally, it focuses on deep learning research in two key areas: metasurface modeling and holographic imaging design. It introduces several deep neural network architectures—the four major classic neural networks, namely CNN, GAN, RNN, and LSTM, all play crucial roles in metasurface modeling and design.

This paper first introduces deep learning-based metasurface design, followed by a discussion on the application of deep learning in metasurface holographic imaging design, with a focus on how these technologies enable rapid design through deep learning. Additionally, the paper provides

Table 1. Advantages, limitations, and most suitable applications of CNN, GAN, RNN, and LSTM models.

Deep Learning Methods	Advantages and Characteristics	Limitations Analysis	Applications
Convolutional Neural Network (CNN)	Spatial Feature Analysis: Well suited for capturing local geometric patterns and spatial correlations of metasurface unit cells. Translation Invariance: Robust to variations in the positions of metasurface units, ensuring consistent performance across different layouts. Parameter Efficiency: The weight-sharing mechanism effectively suppresses the common overfitting issue during the training process. Multi-Scale Analysis: Hierarchical feature maps enable simultaneous capture of unit cell details and array-wide behavior. Fast Inference: Real-time prediction from structure to spectrum, supporting rapid design iterations. Physical Integration: Electromagnetic boundary conditions can be integrated through dedicated convolution kernels.	Sequential Limitation: Cannot directly model frequency-dependent or time-varying metasurface response characteristics. Fixed Input Size: Requires uniform grid discretization, limiting the design flexibility for irregular geometries. Long-Range Dependencies Limited: Struggles to capture global electromagnetic coupling effects in large arrays. Inverse Design Challenge: Excellent performance in forward prediction, but inverse design requires additional optimization algorithm support.	Forward Prediction Spectral Analysis
Generative Adversarial Network (GAN)	Novel Design Generation: Creates entirely new metasurface geometries beyond the training data distribution. Inverse Design Advantage: Directly maps desired optical responses to optimal structural geometries. Design Space Exploration: Discovers non-intuitive structures with superior performance. Multi-Objective Optimization: Simultaneously optimizes multiple electromagnetic properties (efficiency, bandwidth, polarization). Conditional Generation: Can generate designs under specific constraints (manufacturing limitations, material properties). Data Augmentation: Generates synthetic training data to overcome the limitations of small experimental datasets.	Training Instability: Mode collapse and vanishing gradients may limit design diversity. Difficult Convergence: Requires fine-tuning of hyperparameters and balanced generator–discriminator training. Risk of Violating Physical Principles: Generated structures may violate electromagnetic principles or manufacturing constraints. Evaluation Complexity: Difficult to assess the quality of generated designs without full electromagnetic simulations. High Computational Cost: Requires substantial training time and computational resources.	Inverse Design Innovative Exploration
Recurrent Neural Network (RNN)	Sequence Processing: Naturally handles frequency-dependent electromagnetic responses over a wide bandwidth. Time-domain Correlation: Captures dispersion effects and frequency interdependencies in metasurface responses. Variable-Length Inputs: Adapts to varying frequency sampling densities and spectral resolution requirements. Cascaded Design: Models layer-by-layer electromagnetic coupling in multi-layer metasurface structures. Dynamic Metasurfaces: Suitable for time-varying or reconfigurable metasurface designs. Memory Efficiency: Sequence processing reduces memory requirements for large frequency datasets.	Vanishing Gradients: Due to gradient decay, it is difficult to learn long-range frequency dependencies. Sequence Bottleneck: Unable to parallelize between frequency points, leading to slower training and inference speeds. Limited Spatial Understanding: Struggles with processing the 2D spatial relationships of metasurface geometries. Overfitting Tendency: Prone to memorizing specific frequency patterns rather than learning general electromagnetic principles. Context Limitation: Difficult to process very long frequency sequences.	Frequency-Domain Analysis Dispersion Modeling
Long Short-Term Memory Network (LSTM)	Long-Range correlations: Effectively captures broadband electromagnetic responses spanning multiple octaves. Gradient Stability: Gating mechanisms prevent gradient vanishing, supporting deep spectral analysis. Selective Analysis: Forgetting gates filter out irrelevant frequency components, retaining critical resonance information. Dispersion Modeling: Excels at modeling material and geometric dispersion effects. Multi-Band Design: Simultaneously optimizes performance across multiple frequency bands. Strong Robustness: Insensitive to experimental measurement noise and manufacturing variations.	Computational Complexity: The number of neurons inside the model is several times that of a standard RNN, increasing training time and memory requirements. Hyperparameter Sensitivity: Gating bias initialization and learning rates need fine-tuning for electromagnetic applications. Overfitting Risk: High model capacity can easily lead to overfitting on small metasurface datasets. Sequence Processing: Cannot leverage frequency parallelism, making wideband analysis slower than CNNs. Spatial Limitations: Limited ability to capture 2D spatial patterns of metasurface unit cells.	Wideband Design Dispersion Engineering

and

Table 2. Recent advances in metasurface design utilizing CNN, RNN, LSTM, and GAN: A review of academic research progress.

Algorithm	Input	Output	Working Bandwidth	Year
CNN [142]	Structural data	Spectral data	800–1800 nm	2019
CNN [143]	Structural data	Spectral data	10 GHz	2019
CNN [144]	Structural data	Spectral data	8–13 GHz	2019
DC-GAN [145]	Target data	Structural parameters	1–30 GHz	2019
CGAN [146]	Spectral data	Structural pattern	400–600 nm	2021
GAN [147]	Spectral data	Structural data	0.4–1 THz	2021
CNN [148]	Tissue data	Tissue differentiation	1–3 GHz	2022
SLMGAN [149]	Spectral data	Structural image	20–30 GHz	2022
KNN-GAN [150]	Spectral data	Structural image	30–60 GHz	2022
CGAN [151]	Structural data and absorption spectrum	Structural image	80–100 Hz	2022
GAN [152]	Target data	Structural data	4–20 GHz	2022
CNN [153]	Structural data	Spectral data	0.8–1.2 THz	2023
CNN [154]	Structural data	Spectral data	650–780 nm	2023
CNN [155]	Structural data	Spectral data	8–16 GHz	2024
XGAN [156]	Electromagnetic response	Structural image	20–35 GHz	2024
DPN-GAN [157]	Spectral data	Structural image	11.6–24.2 GHz	2024
GAF-CNN-LSTM [158]	2D feature images [159]	Structural image	1340–1400 nm	2024
cDCGAN [160]	Scattering parameters and images	Unit cell image	4–20 GHz	2024
LSTM [161]	Target spectrum	Structural parameters	4.5–5.6 THz	2025
RGAN [155]	Spectral data	Structural image	2–18 GHz	2025
MsCNN-CBAM-LSTM [162]	Structural data	Spectral data	0.2–2.0 THz	2025
CNN [163]	Structural data	Resonant frequency	0.3–1.4 THz	2025
GAF-DCCNN-MHA [164]	2D angular field images [165]	Structural parameters	1300–1600 nm	2025
CGAN [166]	Target acoustic absorption spectrum	Structural image	40–3000 Hz	2025

, which summarize the advantages, limitations, optimal application scenarios, operating bandwidths, and input–output data of the four neural network types (CNN, GAN, RNN, LSTM) used in recent studies.

2. Deep Learning-Based Metasurface Design

Deep learning, inspired by artificial neural networks (ANNs), performs distributed information processing through hierarchical data transformations, mimicking the way biological neural systems work, thereby capturing representative higher-order statistical patterns. Deep learning offers a variety of training mechanisms, which can be adapted to supervised, unsupervised, or self-supervised learning depending on the user’s needs [167]. A typical deep learning model is constructed by stacking L layers of nonlinear transformations, where each layer consists of several differentiable computational units. The computation in each layer can be expressed as ${\mathrm{h}}^{\left(\mathrm{l}\right)}=\mathsf{\sigma }\left({\mathrm{W}}^{\left(\mathrm{l}\right)}{\mathrm{h}}^{\left(\mathrm{l}-1\right)}+{\mathrm{b}}^{\left(\mathrm{l}\right)}\right), \mathrm{l}=1,2,3,\dots ,\mathrm{L}$, where h denotes the output of the l-th layer. The network is defined by its weights (W), biases (b), and nonlinear activation functions (e.g., rectified linear unit, ReLU). Training is achieved through iterative minimization of loss functions (e.g., cross-entropy loss), which determines the parameters of hidden layers and output layers. Physically, deep learning can be viewed as a process where input data undergoes multi-level processing of “linear transformation + nonlinear distortion” across layers, ultimately being converted into a probability distribution. In recent years, the application of neural networks has reached a peak, leading to the emergence of many new algorithms [168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224]. Through computational power, the inherent mechanisms of neural networks can solve problems that previously required significant human and material resources. Examples include image processing, portrait analysis and recognition, image enhancement [225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298], medical case identification and medical data processing [299,300,301,302,303,304,305,306,307,308,309,310,311,312,313,314,315], action recognition [316,317,318,319,320], big data processing [321,322,323,324,325,326,327,328,329,330,331,332,333,334,335,336,337,338,339,340,341,342,343,344,345,346,347,348], and agricultural crop research and related studies [326,349,350,351].

Traditional metasurface design methods mainly include the parameter sweep method and analytical modeling method. The parameter sweep method involves systematically sweeping and optimizing design parameters (e.g., geometric parameters, material parameters, and unit cell arrangements) to achieve the desired electromagnetic response. While intuitive and simple to implement, this approach is computationally intensive and impractical for large design spaces. In contrast, the analytical modeling method constructs analytical models based on specific physical principles (e.g., electromagnetic theory, scattering theory) and mathematical formulas to simulate and analyze the electromagnetic response of metasurfaces. By solving these models, specific parameters and performance metrics can be determined, yielding accurate and rapid simulation results and theoretical guidance. However, it has limitations: simplified models must be constructed for metasurfaces with different design objectives to predict performance optimization. Overall, traditional methods can partially meet design goals but incur significant computational costs when handling high-dimensional or complex scenarios.

Recently, the application of deep learning, a rapidly evolving cross-disciplinary technology, in metasurface design has attracted considerable focus from the research community. This paper summarizes four types of neural network frameworks for designing metasurfaces, and the network framework diagram is shown in Figure 1. By virtue of the superior learning capabilities of neural networks, deep learning can identify complex, hidden relationships between the structural attributes of metasurfaces and their electromagnetic properties, utilizing large-scale datasets to facilitate the design process [352,353,354,355,356,357].

2.1. Development History of Neural Networks

Deep learning is inherently intertwined with deep neural networks, which allows us to trace the birth of this methodology by examining the history of artificial neural networks (ANNs). As shown in Figure 2, one can intuitively understand the key milestones in the development of deep learning. In fact, in 1943, physiologist W. McCulloch and mathematician W. Pitts proposed the renowned MP neuron model [358].

In 1958, AI pioneer F. Rosenblatt developed the first single-layer perceptron model based on biological neuron mechanisms, enabling simple pattern classification through weight learning and laying the foundational framework for neural networks [359]. In 1980, Kunihiko Fukushima proposed the convolutional neural network (CNN) model [360]. In 1982, J. Hopfield introduced the recurrent neural network (RNN) with dynamic stability properties (known as the Hopfield network), which leveraged energy function principles to solve combinatorial optimization problems and expanded the dynamic application scenarios of neural networks [361]. In 1986, G. Hinton’s team developed the Multi-layer Perceptron (MLP) architecture, combining Sigmoid nonlinear activation functions with backpropagation algorithms to achieve gradient-based optimization training for deep networks for the first time, breaking through the limitations of single-layer models in handling nonlinear problems [362]. In 1989, Yann LeCun et al. introduced the standard backpropagation algorithm into deep neural networks [363]. In 1997, J. Schmidhuber’s team developed a novel network that could capture the long-term dependencies in sequential data, known as the Long Short-Term Memory (LSTM) network. This marked the first time that neural networks demonstrated strong modeling capabilities in deep temporal aspects [364]. In 1998, Yann LeCun proposed the first CNN (LeNet) applied to handwritten digit recognition; its alternately stacked convolutional and pooling layer architecture became a foundational design for computer vision models [365]. In the same year, Professor Fei-Fei Li initiated an academic initiative at Princeton University to construct a large-scale image database, which was formally launched as the ImageNet project in 2009, providing critical training resources for deep learning [366].

In 2006, Hinton further integrated probabilistic generative models with Boltzmann machine theory and proposed the Deep Belief Network (DBN), By employing the layer-wise unsupervised pre-training method of Restricted Boltzmann Machines (RBMs), it enabled the extraction of hierarchical features from high-dimensional data, marking the official entry of neural network architectures into the era of depth [367]. In 2012, Alex Krizhevsky’s AlexNet won the ImageNet competition with a 10.9% accuracy margin over the second-place model, first validating the breakthrough performance improvements of GPU acceleration and ReLU activation functions for deep models [368]. In 2013, the VGG group at the University of Oxford proposed VGGNet, which achieved excellent results in the ILSVRC competition through a modular design of continuously stacked 3 × 3 convolutional kernels, confirming the positive correlation between network depth and feature extraction capability [369]. In 2014, Mehdi Mirza et al. introduced supervised information into the unsupervised learning framework of Generative Adversarial Networks (GANs) [370], proposing conditional GANs (cGANs) to apply GANs to supervised learning for generative modeling [371]. In the same year, the Google team successively introduced Inception v1 (adopting multi-scale parallel Inception modules to optimize computational efficiency) and MobileNet (implementing lightweight deployment on mobile devices via depthwise separable convolutions), addressing issues of parameter redundancy and computational resource constraints [372].

In 2016, He Kaiming’s team developed a method that mitigated the long-standing vanishing gradient problem in deep networks through residual skip connections, successfully training a 152-layer network for the first time; its derivative architecture, ResNeXt, enhanced feature reuse efficiency via grouped convolutions [373]. In 2017, Google proposed the Transformer architecture based on self-attention mechanisms, completely replacing RNNs as the mainstream solution for sequence modeling. OpenAI later released the GPT-1 (Generative Pre-trained Transformer) series of pre-trained models in 2018, driving a paradigm shift in natural language processing through unsupervised pre-training and task fine-tuning, ultimately leading to the milestone conversational system ChatGPT (The conversational interaction interface, which is based on GPT-3.5) [368]. Peurifoy and his team, at the same time, developed a new model that uses artificial neural networks to approximate the scattering spectra of multi-layer nanoparticles [374]. Zhaocheng Liu from the University of Georgia proposed a novel method for inverse design of metasurface nanostructures using Generative Adversarial Networks (GANs), aiming to expand the scope of metasurface inverse design [375].

In 2019, Wei Ma et al. proposed a new deep generative model for metamaterial inverse design using Variational Autoencoders (VAEs) and Gaussian Mixture Model (GMM) priors. This model takes the geometric shape of a metasurface as input and outputs a probabilistic representation of its electromagnetic response, while employing semi-supervised training with labeled and unlabeled data to enhance performance [376]. In the same year, Peter R. Wiecha and Otto L. Muskens proposed a Coupled Dipole Approximation (CDA) method combined with deep ANNs. Compared to traditional numerical simulation methods, the trained neural network can more rapidly infer the internal fields of arbitrary three-dimensional nanostructures [377].

In 2020, Zhaocheng Liu et al. proposed a hybrid inverse design strategy, combining Variational Autoencoders (VAEs) and genetic algorithms (GAs), to map all latent design modes of unit structures into a compact space [378]. In the same year, Bowen Zheng et al. proposed a deep learning method that, after training, can accurately predict the amplitude and phase of target units for multi-layer metal metasurface design [379].

In 2021, Christopher Yeung et al. proposed a method to optimize metasurface photonic structures using Conditional Deep Convolutional GANs (cDCGANs) [380]. Zhichao Sun et al. proposed a standardized nearest neighbor algorithm, based on polynomial regression, to design a forward network model, which performs well as a forward model [381]. Ruoqin Yan’s team developed a deep learning method based on recurrent neural networks (RNNs) to extract sequential features from the target data, successfully achieving this effect [382].

In 2022, Mohammadreza Zandehshahvar et al. proposed a manifold learning-based method for metasurface inverse design. This method trains autoencoders (AEs) to compress desired electromagnetic responses into low-dimensional latent space vectors, models different submanifolds within this latent space using GMMs, and optimizes to obtain the optimal submanifold that meets the desired response [383].

In 2023, Yan Teng and his team successfully created a novel forward network based on convolutional neural networks (CNNs), which can perfectly capture the target features [384].

In 2024, to address the inefficiency of manual design processes for traditional cross-polarization conversion metasurfaces, Hongkai Zhou and his team proposed a novel approach that combines electromagnetic simulation software with neural networks to achieve inverse design of cross-polarization conversion metasurfaces [385]. Also in 2024, to address the low efficiency exhibited by traditional metasurface design methods, Yunfei Liu and his team proposed a dual-path network integrated with Generative Adversarial Networks (GANs), which can effectively learn the complex mapping between cross-polarization reflectance and the anisotropic metasurface unit modes [157].

In 2025, driven by the continuous development and expanding demands of microwave technology, Jiayun Wang et al. proposed a GAN with an original network framework (RGAN) to achieve inverse design of metasurfaces from given responses of metasurface modes [155]. Meanwhile, to address the limitation of deep learning in achieving high design freedom for metasurfaces, Huakun Xia and colleagues proposed a design and characterization method for hybrid functional metasurfaces with freeform patterns, based on a deep UNet++ conditional Generative Adversarial Network (GAN). This approach effectively overcomes the challenge of limited design freedom in deep learning-based metasurface design [386].

In recent years, physics-inspired neural networks (PINNs) and their applications in the field of optics have garnered significant attention. By integrating physical laws with machine learning, PINNs overcome the limitations of traditional digital computing and exhibit unique advantages in optics, including high parallelism, low energy consumption, and strong robustness, emerging as a current focal point of research at the intersection of artificial intelligence and optoelectronics. In 2025, a collaborative team from École Normale Supérieure de Paris and Tsinghua University proposed a core scheme summarized as a “coding-scattering-detection” three-step process. This scheme addresses the challenges of next-generation reservoir computing (NGRC), where large-scale polynomial computations incur enormous resource consumption in electronic hardware and are also difficult to directly simulate using most physical systems. Compared to conventional approaches, this method reduces the required training data volume by over 90%, involves fewer hyperparameters, and employs a smaller reservoir size [387].

2.2. Predictive Networks

Predictive networks are a type of discriminative model that identifies explicit category labels or attribute values of objects. It can identify the unique relationship between the input and output during the training process, thereby determining the optimal classification label or attribute value for the given input variable. These networks process input data through feature extraction, feature selection, and other preprocessing steps before using the processed data for learning. The most critical distinction of predictive networks from other network types lies in their sole objective: optimizing classification boundaries. Their training primarily aims to ensure distinct separability between different categories or to maximize the discrepancy between predicted values and actual values. By iteratively adjusting internal network parameters, predictive networks refine their ability to distinguish categories, minimizing the gap between predictions and ground truth, and ultimately generating more precise discriminative boundaries.

In a general sense, The most classic network in predictive networks is the convolutional neural network (CNN), which leverages its inherent properties to hierarchically extract features from input images, transforming them into data that can be recognized by the network.

Convolutional Neural Network (CNN)

The convolutional neural network is one of many deep learning models, where its input layer effectively processes the input data and passes it to the subsequent layers. The spatial dimensions of the raw data critically affect network performance; if the raw data size does not meet network requirements, normalization, resizing, or region cropping is often applied to align the data with network specifications.

The intermediate layers of a CNN consist of three fundamental units: feature extraction units, spatial compression units, and nonlinear transformation units. The feature extraction unit employs a sliding window (i.e., a convolutional kernel) to perform spatial convolution. This window traverses each pixel in the input feature map, computes the dot product between the pixel’s spatial neighborhood and the convolutional kernel to generate feature responses, and aggregates these responses for use in subsequent layers. As the CNN depth increases, convolutional filters combine to form visual features of varying scales. The subsequent spatial compression unit reduces feature map dimensionality through extreme value sampling or mean sampling (e.g., average pooling, computing the average of a local region), which reduces computational complexity and mitigates overfitting. Finally, the nonlinearity transformation unit applies nonlinear activation functions, effectively breaking the limitations of traditional linear models, enabling the network to better model complex feature patterns.

The output layer of a convolutional neural network processes the input data effectively and produces the desired results. In general, for image processing tasks, the number of neurons in the network’s fully connected layer is proportional to the number of target categories. By applying a probability distribution function (e.g., Softmax), the output layer generates a confidence vector for each category, enabling precise image classification [373,388,389,390,391,392,393,394,395,396,397,398,399].

For metasurface design, the convolutional mechanism in CNNs directly models the spatial distribution of metasurface units, allowing the network to learn electromagnetic response features without relying on prior empirical models. Spatial compression layers accelerate computations, enabling real-time optimization of large-scale unit arrays. End-to-end modeling reverse-engineers unit structures that match target optical performance, ensuring process compatibility. The convolutional kernel (filter) of a CNN only connects to local regions (e.g., 3 × 3 or 5 × 5 neighborhoods) of the input feature map, which directly aligns with the requirement for “local geometric feature extraction” of metasurface unit structures. All neurons in the same convolutional layer share the same weight parameters, meaning the model does not need to re-learn parameters for the same local features of each unit. This inherent property of CNNs—local connectivity and weight sharing—demonstrates a strong alignment with the design principles of metasurfaces. Additionally, multi-channel processing facilitates the co-optimization of multiple physical fields, enhancing both optical performance and structural stability/process adaptability [154,378,400,401,402,403,404,405,406,407,408,409,410,411,412,413]. The framework diagram of the partial convolutional neural network for metasurface design is shown in Figure 3. Here, Figure 3a illustrates the process of predicting the absorption curve from the base structure via model-based prediction [142]; Figure 3b demonstrates the process of predicting spectral data by training the model with metasurface structural parameters [414]; and Figure 3c depicts the process of predicting the real and imaginary components of the metasurface through model training [400].

2.3. Generative Networks

Generative networks aim to comprehensively understand the intrinsic probability distribution characteristics of original training datasets and generate new data samples that adhere to this distribution. The principle of generative networks is based on the probabilistic distribution relationships in the input data or the distribution of generative probabilities. By modeling joint distributions, generative networks represent the dependency between the probability of each sample point in X and the corresponding value in Y. Additionally, generative networks incorporate latent variable modeling to better capture the topological structure and density distribution information of the data manifold. Unlike predictive networks, whose primary goal is to seek classification boundaries, generative models strive to uncover the entire data space distribution. This property enables generative networks to transform random noise into high-quality samples.

The most representative generative model is the Generative Adversarial Network (GAN) [415,416,417,418,419,420,421,422,423,424,425,426], optimized through an adversarial training process between a generator (G) and a discriminator (D). In generative networks, the role of the generator is to produce fake samples that closely resemble real samples, while the corresponding discriminator’s role is to distinguish between real and fake samples. After multiple iterations, the generator will produce samples that closely approximate real ones. The continuous cycle of outputting data and improving discriminative ability represents an ongoing effort to enhance sample quality.

Generative Adversarial Network (GAN)

In GANs, the input interface (G) receives signals from the latent space, mapping low-dimensional randomly distributed latent variables (typically sampled from Gaussian or uniform distributions) to high-dimensional, complex pseudo-data samples.

The number of latent variables and their distribution determine the quality and diversity of generated samples, with these variables parameterized and optimized through differentiable mappings. The selection and distribution of the quality of generated samples is important; optimizing latent space sampling thus facilitates the generation of superior samples.

The generator, which creates fake samples, and the discriminator, which distinguishes between real and fake samples, together form a Generative Adversarial Network. The entire network is refined through an adversarial “game,” where the generator employs spatial transformation layers to perform nonlinear upscaling of latent variables. After each feature transformation, it applies learnable linear transformation layers and nonlinear activation functions to convert random latent variables into pseudo-samples that exhibit statistical characteristics of the target data domain. The generator’s core objective is to produce pseudo-samples that are indistinguishable from real samples, such that the discrepancy between them falls below the discriminator’s detection threshold.

The discriminator performs a binary classification task: given an input sample (either real or generated by the generator), it uses a spatially sensitive feature-judging unit (e.g., a CNN or deep fully connected network) to determine its authenticity. During training, the discriminator receives one input real sample and another fake sample generated by the generator, which is similar to the real sample. Through hierarchical feature extraction layers (e.g., convolutional or fully connected layers), it learns spatial or semantic correlations in the data and outputs a single judgment (typically using a Sigmoid function). The ultimate goal of the discriminator is to correctly identify which samples are real and which are fake samples generated by the generator.

Generator and discriminator operate within a zero-sum game framework, where they are alternately optimized. The goal of the generator is to produce samples that increasingly resemble real samples, effectively searching for a latent variable distribution z that minimizes this detection probability. The goal of the discriminator is to correctly identify which samples are real and which are fake samples generated by the generator, and to constrain the model through the loss function in a backward manner, both components adjust their parameters iteratively, driven by minimax game theory, to approximate the true data distribution of the training set.

As the final component of the generator, the output layer must produce samples that closely resemble the features of the real dataset. In image generation tasks, the generator uses transposed convolutions or upsampling layers to ensure the output images match the dimensions and pixel matrix size of real images. For tasks such as data augmentation or style transfer, the generator produces sample sets with specific characteristics (e.g., a particular style or category) while achieving sufficient realism.

In metasurface design, GANs address bottlenecks in electromagnetic simulation software by directly mapping latent variable geometries to corresponding optical responses in an end-to-end manner. For instance, GANs can rapidly generate subwavelength structures with dozens of degrees of freedom in a fraction of the time required by traditional methods. Beyond speed, GANs enable exploration of high-dimensional parameter spaces, facilitating the discovery of metasurface structures that satisfy optical functional requirements. By incorporating target functions (e.g., phase distributions or transmission spectra) as conditional inputs, GANs avoid common pitfalls of traditional optimization methods (e.g., local optima) and incorporate electromagnetic performance constraints, achieving global optimization in a single training cycle.

The core objective of metasurface inverse design is as follows: given a target electromagnetic response (e.g., a specific phase response in a frequency band), generate a metasurface structure that satisfies the requirement. However, this problem inherently exhibits a “one-to-many” characteristic—a single target response may correspond to infinitely many physically feasible metasurface structures (e.g., different combinations of unit geometric parameters may achieve the same phase response). Traditional supervised learning methods (e.g., CNN regression) typically learn only one “average solution,” which fails to cover the diversity of the solution space and results in design outcomes that are singular and lack innovativeness. The adversarial training mechanism of GANs (generator–discriminator game) addresses this issue by forcing the generator to learn the solution space distribution corresponding to the target response—to deceive the discriminator, the generator must not only produce a single “optimal solution” but also cover a broader range of “feasible solutions” (e.g., different unit sizes that still match the phase requirements). This mechanism enables GANs to output diverse metasurface structures, significantly enhancing the innovativeness of inverse design. Among them, the framework diagrams of metasurface design using GAN models in recent studies are shown in Figure 4. Figure 4a illustrates the architecture of the discriminator and generator in the DPN-GAN model [157]. Figure 4b demonstrates the overall workflow and framework of the Deep UCGAN+ model [386]. Figure 4c outlines the reverse design process of the SLM-GAN model [149]. Figure 4d presents the framework and operational workflow of the XGAN model [427].

2.4. Sequential Networks

Sequential networks aim to gain insights into the temporal dependencies of input data in order to infer the subsequent evolutionary process of the data. Their core focus lies in learning the conditional probability distribution within sequences, where the state at the current time step is a function of past states, thus being characterized as a conditional probability. Unlike traditional feedforward models, sequential networks incorporate recurrent connections that capture historical dependencies, endowing them with dynamic memory capabilities.

During training, sequential networks employ recursive methods for sequence learning. At each time step, the network self-adjusts based on the input data to transmit the historical information of the data into subsequent training. This forms a dynamic modeling framework for handling temporal dependencies, with the recurrent network (RNN) being the most representative early example. RNNs use a hidden state variable that, at each time step, processes the feature information of the current input together with the feature information from the previous step, and then updates itself, thereby accumulating and propagating historical information to establish the foundation for sequence modeling [428,429].

However, Basic RNN models fail to address the vanishing gradient problem, which greatly limits the model’s learning ability. It was precisely to address the gradient constraint that the unique gating mechanism was introduced, leading to the emergence of the Long Short-Term Memory (LSTM) model. These gates enable LSTM to effectively manage long-term dependencies, overcoming the learning limitations of traditional RNNs [430,431,432].

Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM)

The input layer of an RNN receives sequential data from external sources and passes the temporal order and scale distribution of each sequence element to subsequent processing layers. The quality of RNN performance depends heavily on the temporal order and scale distribution of the raw input data. Preprocessing (e.g., handling variable sequence lengths or noise) is often required to facilitate further computation in downstream layers. Owing to its distinctive intrinsic mechanisms, the RNN has been integrated to varying extents into diverse domains, including medicine [156] and architecture [433].

The intermediate layers of an RNN serve as the primary computational engine, with their defining feature being the use of temporal recurrence. A hidden state unit is used to store relevant information, and based on this feature, it effectively combines current information with previous information. After linear transformation and nonlinear activation (e.g., Tanh, ReLU), this combined information updates the current hidden state, forming a compressed dynamic representation of sequence information. The previous hidden state serves both as the input data for the current step and propagates information to subsequent training, which enables the network to learn by relying on long-term dependencies. However, standard RNNs are helpless against the vanishing gradient problem because they lack the ability to capture long-term dependencies, thus failing to address the issue when processing sequences. In metasurface design, this limitation complicates the learning of far-field interference effects between non-adjacent components. LSTM addresses this by introducing triple gating mechanisms and redesigning the memory cell to create an “information highway” via a cell state [434,435,436,437,438,439]. This allows phase errors to propagate from hundreds of units to the network’s output, resolving the vanishing gradient bottleneck.

Additionally, incorporating nonlinear transformation units (e.g., Tanh, ReLU) into the state update process enables the network to transcend simple linear mapping, facilitating more complex pattern recognition. The network’s output layer consolidates the overall decisions and information: for each time step, the hidden state is processed through the corresponding layer to generate a decision or class vector. In sequence prediction tasks, these vectors are concatenated across time steps to form the final output. For sequence classification/labeling tasks, outputs from each time step are aggregated, relevant information is extracted, and the result is mapped to a probability distribution (via Softmax or similar functions) for class determination. Sequence-to-sequence tasks simplify this by using an encoder–decoder architecture. In order to facilitate subsequent processing, the encoder is used to extract relevant information.

The recurrent structure of RNNs aligns with the spatial ordering characteristics of metasurface unit arrays, effectively leveraging hidden states to model near-field coupling effects (e.g., near-field interference) between metasurface units. This addresses the limitation of conventional segmented designs, which often neglect long-range correlations. The cell state in LSTM further enhances the network’s ability to efficiently describe coupling effects between distant units via transmission and diffraction, enabling global phase matching across units separated by hundreds of micrometers.

For broadband or wide-wavelength multi-wavelength optimization, wavelengths can be discretized into a sequence, with cross-band phase constraints accumulated in the hidden state memory. LSTM’s input gate dynamically adjusts weights for different wavelengths, resolving inter-band interference while optimizing the global solution. In inverse design, RNN decoders can iteratively generate geometric parameter sequences that satisfy target spectral responses (e.g., reflectivity curves). Process constraints, such as lithography resolution, can also be incorporated to ensure design manufacturability. Owing to their strong long-sequence learning capability, RNNs are well-suited for end-to-end modeling, directly generating non-periodic metasurface gratings or other non-periodic heterogeneous topological structures. As shown in Table 1, one can more intuitively understand the differences among these four models, while Table 2 provides the working bandwidths, input–output data of papers using these models in recent years, as well as the publication times of the papers. Additionally, the RNN model frameworks used in some of these papers are shown in Figure 5. Figure 5a illustrates the architecture and training workflow of the GAF-CNN-LSTM model [158]. Figure 5b demonstrates the structural design of the CNN-LSTM-A model [440]. Figure 5c provides a schematic representation of the deep learning methodology integrating Feature Pyramid Networks (FPNs) and Inverted Residual Networks (IDNs) [161]. Figure 5d presents the architectural framework of the NanoPhotoNet model [441].

3. Holographic Applications of Deep Learning-Designed Metasurfaces

Metasurface holographic imaging leverages the precise manipulation of electromagnetic wavefields by artificial subwavelength structures. It involves designing the spatial arrangement and geometric layout of metasurface arrays to modulate various parameters of light waves [169,227,442], based on the digitized target optical field distribution, thereby achieving holographic imaging. Metasurfaces offer significant advantages over traditional optical components in terms of compactness and multi-dimensional multifunctionality, with unit volumes capable of supporting broader fields than planar microlenses [163,164,165,166]. This is attributed to the localized electromagnetic resonance phenomena induced by subwavelength-scale artificial unit cells, which generate additional localized effects. By exploiting this localized enhancement, it can be used to adjust the parameters of light waves in metasurfaces.

Holographic imaging records the true scattered light field of a target object and relies on the interference of two or more coherent light beams to reconstruct the image.

In contrast, non-holographic imaging computes the diffraction field generated by a known target wavefront under specific conditions to extract partial object information, which is then encoded into a complex matrix. Metasurface holography combines the strengths of both approaches: it designs the target optical field distribution as a phase modulation function for each unit in reverse order, converts continuous phase profiles into digital coding patterns, and uses the arrangement of unit structures to form the encoded region of the hologram. When illuminated by an incident light wave, the metasurface device generates localized phase mutations across its subwavelength elements, mimicking the effect of a hologram—focusing light at predetermined positions to reconstruct the target optical field distribution as the metasurface wavefront. Upon reconstruction illumination, the diffracted light wave, modified by the encoded metasurface, reproduces the previously recorded information, effectively recreating the target optical field distribution. This mechanism of “digital design—nanostructure encoding—wavefront reconstruction” replaces traditional holography’s physical objects and complex interference systems with precise control over imaging position, depth of field, and multi-channel display via tailored metasurface unit arrangements, making it suitable for dynamic holographic display, optical information encryption, and other applications. The model frameworks for designing metasurfaces using neural networks, applied in holographic imaging in recent studies, are shown in Figure 6. Figure 6a illustrates the end-to-end design architecture and operational workflow for metasurface-based holographic imaging [89]. Figure 6b demonstrates the architectural framework of the convolutional neural network (CNN) implementing holographic imaging [96]. Figure 6c outlines the data processing pipeline of the CODE model [115].

Deep learning excels at learning latent patterns from large datasets and predicting metasurface structures that meet holographic imaging requirements in extremely short times, enabling efficient and automated design of deep learning-driven metasurface holograms [443]. The natural advantages of deep learning in holographic imaging have attracted significant research attention, yielding landmark achievements. For example, in 2022, Wei Wei et al. proposed a physics-constrained unsupervised deep learning framework for designing metasurface complex amplitude holographic devices under incomplete optical modulation conditions [444]. In the same year, Kang Wang et al. expanded the structural diversity of deep learning-designed metasurfaces by using deep learning to design origami metasurfaces for successful hologram imaging [445]. To address the challenging problem of inversely designing metasurface geometries for given target optical responses, Jierong Cheng et al. introduced an unconventional spectral-driven inverse design method based on deep convolutional architectures in 2023. This network designs unit cells with user-defined phase delays under different frequencies and polarizations, satisfying diverse requirements for multifunctional waveguides and expanding the search space [153]. Meanwhile, Sixue Chen and his team developed a novel model framework, distinct from other frameworks, that can effectively improve the efficiency of metasurface inverse design, achieving inverse design of chiral metasurfaces. The framework successfully constructed metasurfaces that perform phase modulation via Pancharatnam-Berry (PB) phases for right-handed circularly polarized waves at terahertz frequencies, enabling focused vortex beam generation by metasurface arrays [446]. Additionally, Chia-Hsiang Lin et al. addressed the large size of traditional hyperspectral imaging systems by proposing a hybrid deep learning model based on a Transformer + CNN architecture. Using a CODE-driven imaging system, this framework efficiently generates high-fidelity 18-band hyperspectral data cubes with only 18 training data points [447]. In 2024, Qiang Weng and his team incorporated a multi-objective inverse adjoint architecture into previous frameworks to effectively overcome the limitations of traditional methods. This novel framework significantly reduces the risks associated with variable correspondence in conventional neural network-based inverse design. The technique shows promising potential for actively adjusting color images under different environmental conditions, offering new solutions for related fields [448]. To improve holographic imaging quality and efficiency, Zheyu Hou and his team, at the same time, developed a deep learning framework for the on-demand design of holographic metasurfaces. This computational model inversely designs all-silicon terahertz metasurfaces, successfully reproducing two preset patterns with an average absolute error (MAE) of 0.015 in imaging precision, significantly enhancing design efficiency and imaging quality [449].

Figure 6. Models in deep learning for metasurface holographic imaging applications. (a) Network architecture for end-to-end design using the complex amplitude hologram of the metasurface [444]. (b) Structure of the convolutional neural network [153]. (c) Training flowchart of an innovative theoretical framework fusing Convex Optimization (CO) and deep learning (DE) [447].

Figure 6. Models in deep learning for metasurface holographic imaging applications. (a) Network architecture for end-to-end design using the complex amplitude hologram of the metasurface [444]. (b) Structure of the convolutional neural network [153]. (c) Training flowchart of an innovative theoretical framework fusing Convex Optimization (CO) and deep learning (DE) [447].

4. Conclusions

This review thoroughly explores four classical deep learning models—CNNs, GANs, RNNs, and LSTMs—and their applications in metasurface design and holographic imaging. The integration of deep learning has rendered metasurface design more diverse, rapid, and precise, simplifying the previously complex inverse design process.

Deep learning models in metasurface design face significant challenges due to their high dependency on the quality and coverage of training data, yet obtaining high-quality data in metasurface design is inherently difficult. This issue is addressed by generating “quasi-realistic datasets” through geometric perturbations (e.g., randomly adjusting unit dimensions by ±5%) and material parameter perturbations (e.g., dielectric constant fluctuations of ±10%) on existing simulation data, thereby enhancing model robustness. Additionally, mixed data training and unsupervised/self-supervised learning are incorporated to further mitigate data limitations.

To effectively embed physical constraints (e.g., manufacturing rules, material losses) into models, manufacturing constraints (e.g., minimum feature size, layer thickness) are converted into mathematical constraints. These constraints are integrated via conditional Generative Adversarial Networks (cGANs) (where constraints are input as conditions to the generator) or by incorporating constraint discriminant branches into discriminators (e.g., to determine if a structure is manufacturable).

Meanwhile, to balance the high computational costs of early-stage model training/data generation with the demands for fast and accurate inference in later stages, two key strategies are employed: (1) model lightweighting—reducing parameter scales through techniques such as pruning, quantization, and knowledge distillation; and (2) hierarchical modeling—using lightweight models to generate initial candidate structures, followed by fine-grained models to optimize details. These approaches collectively balance global efficiency (training/inference speed) with local precision (design accuracy).

Thus, the intersection of deep learning and metasurface design engineering represents a novel and promising direction, aiming to enhance the precision and speed of traditional metasurface design. This paper summarizes the applications of deep learning-designed metasurfaces in holographic imaging. Future work should extend this technology to other related fields, For instance, in multi-physics collaborative design—COMSOL-based electro-thermal coupling models are employed to simulate Joule heating effects in metasurfaces, combined with TensorFlow-trained neural networks for temperature field prediction, and genetic algorithms are utilized to optimize unit cell geometric parameters. For dynamic metasurface real-time control—leveraging the photoconductive properties of perovskite materials and edge computing for signal processing optimization, 0.1 ms-level beam steering is achievable. End-to-end design eliminates error accumulation in traditional step-by-step workflows. In end-to-end full-system design—a joint optimization model integrating “target requirements—metasurface structure—array layout” is constructed, where diffusion models generate initial structures, followed by reinforcement learning for parameter optimization. The potential of emerging neural network architectures—Graph Neural Networks (GNNs) and diffusion models—offer new paradigms for metasurface design. Additionally, GNN-based modeling of electromagnetic coupling between unit cells, combined with physics-constrained loss functions, enables the design of multi-band operational metasurfaces. These aspects collectively highlight the diversity of future research directions.