Generative Adversarial Optical Networks Using Diffractive Layers for Digit and Action Generation

Abstract

Within the traditional electronic neural network framework, Generative Adversarial Networks (GANs) have achieved extensive applications across multiple domains, including image synthesis, style transfer and data augmentation. Recently, several studies have explored the use of optical neural networks represented by the diffractive deep neural network (D²NN) for GANs. However, most of these focus on applications of the generative network, and there is currently no well-established D²NN architecture that simultaneously implements generative adversarial functionality. Here, we propose a novel implementation scheme for generative adversarial networks based on all-optical diffraction layers, demonstrating a complete all-optical adversarial architecture that simultaneously realizes both the generative network and the adversarial network (D²NN-GAN). We validated this method on the MNIST handwritten digit dataset, achieving Nash equilibrium convergence with the discriminator accuracy stabilizing around 50%. Concurrently, the average SSIM parameter of generated images reached 0.9573, indicating that the generated samples possess high quality and closely resemble real samples. Furthermore, we extended the framework to the KTH human action dataset, successfully reconstructing the “running” action with a discriminator accuracy of approximately 75%. The D²NN-GAN architecture introduces a fully optical generative adversarial model, providing a practical path for future optical modeling methods, such as image generation and video synthesis.

1. Introduction

The exponential growth of deep neural networks has revolutionized artificial intelligence, enabling unprecedented capabilities in image recognition, natural language processing and generative modeling [1,2,3]. However, this remarkable progress has come at a substantial computational cost, with state-of-the-art models requiring massive computational resources that strain the limits of conventional electronic processors [4,5]. The energy consumption of training large-scale neural networks has become a critical concern, with some models consuming as much electricity as small cities during their training phase [6,7]. This computational bottleneck has motivated the exploration of alternative computing paradigms that can overcome the fundamental limitations of electronic architecture.

Optical neural networks (ONNs) have emerged as a promising solution to address these challenges, leveraging the inherent parallelism and energy efficiency of photonic information processing [8,9,10,11,12,13,14]. Light propagation through optical media naturally performs matrix multiplications at the speed of light with minimal energy dissipation, making it ideally suited for neural network computations [15,16]. Recent breakthroughs in integrated photonics have demonstrated optical implementations of various neural network architectures, including convolutional neural networks [17], recurrent neural networks [18] and reservoir computing systems [19].

Among various optical neural network implementation schemes, Diffractive Neural Networks (D²NNs) have garnered significant attention due to their advantages of large-scale computational matrices, low loss and high parallelism [20,21,22,23,24,25]. D²NNs consist of successive diffractive layers that modulate the phase and/or amplitude of incident light, with each layer containing trainable neurons arranged in a two-dimensional array. The network parameters are encoded in the transmission coefficients of these layers, which can be physically realized using various technologies, including 3D-printed diffractive elements [26], spatial light modulators [27] and Reconfigurable/Plug-and-Play Meta surfaces [28]. D²NNs offer several compelling advantages: they perform inference at the speed of light, consume no power except for the input illumination and can process entire images in parallel without scanning or serialization [29,30].

Generative adversarial networks (GANs), introduced by Goodfellow et al. [31], represent one of the most significant innovations in machine learning over the past decade. GANs employ two neural networks in an adversarial game: a generator that creates synthetic data and a discriminator that distinguishes real from fake samples [32]. This adversarial training process has proven remarkably effective for generating high-quality synthetic data, with applications ranging from photorealistic image synthesis [33,34,35] to drug discovery [36] and material design [37]. The success of GANs stems from their ability to capture complex data distributions without explicit likelihood estimation, learning through a minimax game that pushes both networks toward improved performance [38,39].

Recently, a groundbreaking study demonstrated optical generation of Van Gogh-style paintings using diffractive deep neural networks (D²NN) [40]. This work reported” optical generative models” by distilling a digital diffusion model into an optical generation pipeline, experimentally demonstrating snapshot generation including Van Gogh-style artworks under monochrome and multiwavelength illumination. In parallel, Zhan et al. demonstrated photonic diffractive generators by sampling optical noises from scattering media [41]. Moreover, Qiu et al. proposed an optoelectronic GAN (OE-GAN) that distributes the generator and discriminator across optical and electronic components, using a diffractive-network-based optoelectronic generator with an electronic MLP discriminator [42]. These studies delved deeply into the potential applications of D²NN and generative networks. However, to our knowledge, no research has yet successfully implemented a complete GAN architecture—including both optical generator and optical discriminator—using diffractive optical networks.

Here, we present a strategy for implementing generative adversarial networks employing two five-layer D²NN networks—one as the generator and the other as the discriminator. By using the MNIST handwritten digit dataset, the generator network transforms spatially encoded Gaussian beams with varying positions and waist radii into synthetic digit patterns. The generated samples are mixed with real images from the original dataset and fed into the discriminator network. The output layer of the discriminator network is divided into two regions, classifying samples as real or fake based on the detected light intensity. In a generative adversarial network, the training cycle alternates between the generator network and the adversarial network. Our simulation results demonstrate that D²NN-GAN successfully achieves Nash equilibrium: the discriminator’s accuracy converges to approximately 50% (The generator’s loss value cannot be reduced further, as the images it produces are already sufficiently realistic. At this point, the discriminator can only perform blind tests, randomly determining whether an image is real or fake). Furthermore, the generator network achieves an average SSIM index of 0.9573 and an FID index of 106.31—indicating that the generated images approach the quality level of real samples.

Building upon this foundation, we further extended the framework to the more challenging KTH Human Action Dataset. Through generative adversarial training on the “running” action, D²NN-GAN ultimately achieved a classification accuracy of 75%. The generated images attained an average SSIM index of 0.2658 and an FID index of 253.71. The failure to achieve Nash equilibrium with an accuracy rate of 50% is primarily attributed to the increased data complexity in the KTH dataset compared to MNIST (expanding from 28 × 28 pixel images to 120 × 160 pixel images). Therefore, we validated a diffractive network with 400 × 400 neurons after expanding the network scale. The average SSIM index of generated images improved to 0.4665, while the FID index decreased to 204.81, achieving a more effective D²NN-GAN.

Diffractive neural network-based fully optical generative adversarial networks provide a practical fully optical generative adversarial approach. Our results demonstrate that this approach exhibits strong generative adversarial capabilities across both handwritten digit and human motion datasets. Leveraging multiple advantages of diffraction networks—such as ultra-fast generation and inference at minimal energy consumption, along with robust parallel processing capabilities that may support multi-sample parallel processing in the future [43]—D²NN-GAN holds promise for effective application across diverse image generation scenarios.

2. Methods

2.1. Forward Propagation Model of D²NN

Our D²NN -GAN framework consists of two independent D²NNs operating in cascade. Each network comprises five diffractive layers separated by free-space propagation distances of 40λ, with each layer containing 200 × 200 trainable neurons arranged in a square lattice with a pixel pitch of 0.5λ. The networks operate at a wavelength of λ = 1550 nm, chosen for its availability in coherent light sources and compatibility with standard optical components.

The forward propagation through each diffractive layer is modeled using the Rayleigh-Sommerfeld (RS) diffraction integral [44]:

$$U_{l+1}\left(x_{l+1},y_{l+1}\right)=\iint U_l\left(x_l,y_l\right)·t_l\left(x_l,y_l\right)·h\left(x_{l+1}-x_l,y_{l+1}-y_l\right)d{x}_{l}d{y}_l,$$

(1)

where $U_l$ represents the complex field at layer l, $t_l$ denotes the complex transmission function of layer l, and h is the free-space transfer function given by:

$$h\left(x,y\right)={\displaystyle \frac{1}{i\lambda }}{\displaystyle \frac{\mathit{exp}\left(ikr\right)}{r}}cos\theta ,$$

(2)

with $r=\sqrt{x^2+y^2+z^2}$ and $cos\theta =z/r$. In the formula, $i^2=-1$. The transmission function of each neuron is parameterized as $t=exp\left(i\varphi \right)$, where $\varphi \in [0,2\pi ]$ represents the trainable phase modulation. The RS is a rigorous scalar propagation model (not a paraxial Fresnel approximation) under the scalar-wave assumption. Moreover, the RS formulation is mathematically equivalent to the angular spectrum method, i.e., a real-space convolution with the free-space Green’s function corresponds to a Fourier-domain transfer function $\mathit{exp}\left(ikr\right)$. Under our propagation distance (40λ) and discrete layer sampling, evanescent spectral components decay exponentially with z and therefore do not contribute appreciably at the subsequent layer. These considerations justify the physical consistency of the RS-based forward model used throughout this study. In our work, backpropagation is performed numerically on a computer, and the trained phase masks are then used for optical inference/generation.

2.2. Gaussian Beam Parameters

The generator network transforms structured input light into synthetic digit patterns. We encode the input using Laguerre-Gaussian (LG) beams, which provide a rich basis for spatial mode representation [45]:

$$\begin{array}{ll}{LG}_{p,l}\left(r,\phi ,z\right)=& \sqrt{{\displaystyle \frac{2p!}{\left(\pi \left(p+\left|l\right|\right)!\right)}}}{\displaystyle \frac{1}{w\left(z\right)}}{\left({\displaystyle \frac{r\sqrt{2}}{w\left(z\right)}}\right)}^{\left|l\right|}{L}_p^{\left|l\right|}\left({\displaystyle \frac{2r^2}{w^2\left(z\right)}}\right)\mathit{exp}\left(-{\displaystyle \frac{r^2}{w^2\left(z\right)}}\right)\mathit{exp}\left(il\phi \right)\\ & \mathit{exp}\left(ikz\right)\mathit{exp}\left(i\left(2p+\left|l\right|+1\right)\zeta \left(z\right)\right),\end{array}$$

(3)

where p and l are the radial and azimuthal mode indices, w(z) is the beam waist, $L_p^{\left|l\right|}$ represents the associated Laguerre polynomial, and $\zeta \left(z\right)=arctan(z/zR)$ is the Gouy phase with Rayleigh range $z_R=\pi w_0^2/\lambda$.

For our implementation, we use fundamental Gaussian modes (p = l = 0) with varying beam waists ranging from 25λ to 35λ and lateral positions spanning a 3 × 3 grid across the input plane. This parameterization provides sufficient diversity in the input space while maintaining Simulation feasibility.

2.3. Discriminator Network Parameters

The discriminator network receives either real digits from the MNIST dataset or synthetic digits from the generator. The input images are encoded as amplitude distributions. The discriminator’s output plane is divided into two square regions: a “True” region centered at $(x_T,y_T)=$(−23.25 $\mathsf{\mu }\mathrm{m}$, 0) and a “False” region at $\left(x_F,y_F\right)=$ (23.25 $\mathsf{\mu }\mathrm{m}$, 0). The area diameter is 15.5 $\mathsf{\mu }\mathrm{m}$. The classification decision is based on the integrated intensity in each region:

$$I_{True}={\iint }_{RT}{\left|U_{out}\left(x,y\right)\right|}^{2}dxdy,$$

(4)

$$I_{False}={\iint }_{RF}{\left|U_{out}\left(x,y\right)\right|}^{2}dxdy,$$

(5)

where I represents the intensity, RT and RF denote the detected square regions. The category represented by the higher intensity region in the two square areas is the optical inference classification.

2.4. Network Training Parameters

We implement alternating gradient descent with the Adam optimizer [46], using learning rates of ${\alpha }_G=$ 0.001 for the generator and ${\alpha }_D=$ 0.0005 for the discriminator. The asymmetric learning rates help stabilize training by preventing the discriminator from overwhelming the generator. As stated in the main text, the discriminator network is solely responsible for recognizing and inferring the intensity distribution within ten designated rectangular regions, while the generator network must handle full-image generation. The differing task complexities dictate these hyperparameter distinctions. If both networks share the same learning rate, gradient training will reference both rates simultaneously. Since the discriminator network has already reached an optimal state, its loss value becomes difficult to further reduce. This prevents the generator network’s loss from decreasing further, causing it to converge to a local optimum. All models in this work were constructed and trained based on PyTorch 2.1.2, implemented within a Python 3.9 computational framework. The simulations were executed on a 64-bit workstation equipped with an Intel Core i9-10940k CPU, NVIDIA GeForce RTX 4080 Ti GPU, and 256 GB DDR4 system memory.

Each training batch consists of 8 real samples and 8 generated samples, with the discriminator updated twice for each generator update to maintain competitive balance. It is important to note that selecting a batch size within an appropriate range will not affect the overall generative and discriminative quality of the network. This is because the number of parameters in diffractive neural networks (approximately 200 × 200 × 5 = 0.2 million) is significantly smaller than that of convolutional neural networks like VGG-16 (approximately 138 million parameters). Thus, batchsize exerts minimal influence on the overall gradient training process.

2.5. FID Calculation Method

To quantitatively evaluate the quality and diversity of generated images, we employed the Fréchet Inception Distance (FID) as a comprehensive metric. FID measures the similarity between the distributions of real and generated images by comparing their feature representations in a high-dimensional embedding space. Unlike pixel-level metrics, FID captures both the perceptual quality and diversity of generated samples.

2.5.1. Feature Extraction with Inception-v3

The FID calculation relies on a pre-trained Inception-v3 network to extract high-level features from images. The Inception-v3 model, trained on ImageNet, serves as a feature extractor that maps images to a 2048-dimensional feature space. For our implementation, we modified the standard Inception-v3 architecture by removing the final classification layer and extracting features from the last average pooling layer.

Since our diffractive networks generate single-channel images (grayscale) with dimensions of 200 × 200 pixels for MNIST and 160 × 120 pixels for KTH dataset, we applied the following preprocessing steps:

1. Channel replication: Single-channel images were replicated across three channels to match the RGB input format expected by Inception-v3: IRGB(x, y) = [Igray(x, y), Igray(x, y), Igray(x, y)]

2. Spatial resizing: Images were resized to 299 × 299 pixels (the standard Inception-v3 input size) through zero-padding: Ipadded (x,y) = pad(IRGB (x,y), (hpad, wpad)), where hpad = ⌊(299 − Horiginal)/2⌋ and wpad = ⌊(299 − Woriginal)/2⌋

3. Intensity normalization: Pixel intensities were normalized to the range [0, 1]: Inorm(x,y) = (Ipadded(x,y) − min(Ipadded))/(max(Ipadded) − min(Ipadded) + ε), where ε = 10 − 8 ensures numerical stability.

The forward propagation through the modified Inception-v3 network can be expressed as: f = Inception-v3pool(Inorm), where f ∈ R2048 represents the feature vector for each image.

2.5.2. Statistical Computation

For a set of N images, we computed the activation statistics by passing all images through the feature extractor in batches to obtain the feature matrix F ∈ RN × 2048. The mean vector and covariance matrix of the feature distribution were calculated as:

$$\mu ={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{f}_i,$$

(6)

$$\sum =({\displaystyle \frac{1}{N-1}}){\sum }_{i=1}^N(f_i-\mu ){(f_i-\mu )}^T,$$

(7)

where f_i denotes the feature vector of the i-th image, μ ∈ R2048 is the mean vector, and Σ ∈ R2048 × 2048 is the covariance matrix.

2.5.3. Fréchet Distance Calculation

The FID score quantifies the distance between two multivariate Gaussian distributions by computing the Fréchet distance (also known as the Wasserstein-2 distance) between them. Given the mean vectors (${\mu }_{real}$, ${\mu }_{fake}$) and covariance matrices (${\sum }_{real}$, ${\sum }_{fake}$) for real and generated image features, the FID is calculated as:

$$FID=\parallel {\mu }_{real}-{\mu }_{fake}{\parallel }^2+Tr({\sum }_{real}+{\sum }_{fake}-{2({\sum }_{real}·{\sum }_{fake})}^{{\displaystyle \frac{1}{2}}}),$$

(8)

where $\parallel$·${\parallel }^2$ denotes the L2 norm, Tr(·) represents the matrix trace, and ${({\sum }_{real}·{\sum }_{fake})}^{{\displaystyle \frac{1}{2}}}$ is the matrix square root of the product ${\sum }_{real}·{\sum }_{fake}$.

The first term $\parallel {\mu }_{real}-{\mu }_{fake}{\parallel }^2$ measures the squared Euclidean distance between feature means, capturing the difference in average characteristics. The second term $Tr({\sum }_{real}+{\sum }_{fake}-{2({\sum }_{real}·{\sum }_{fake})}^{{\displaystyle \frac{1}{2}}})$ quantifies the difference in feature distributions, accounting for both variance and correlations within each distribution.

3. Results

3.1. GAN Principle Based on D²NN

Figure 1 illustrates the complete architecture of our diffractive GAN system. The generator network receives Gaussian beams with varying spatial parameters and transforms them through five diffractive layers to produce synthetic digit patterns. These generated samples are then mixed with real MNIST digits (or human movement digits) and fed to the discriminator network (bottom), which performs binary classification through spatially separated output regions. The adversarial feedback loop drives both networks toward improved performance, with the generator learning to produce increasingly realistic digits while the discriminator becomes more sophisticated at detecting fakes.

3.2. D²NN-GAN for the MNIST Dataset

Figure 2 presents the learned phase distributions and optical field evolution through both networks after training convergence. Figure 2a,b shows the phase patterns of the five diffractive layers for the generator and discriminator networks, respectively. The Gaussian beam collectively transforms into complex digit morphologies. In contrast, the discriminator’s phase patterns (Figure 2b) display more uniform distributions with subtle variations, indicating a focus on feature extraction rather than pattern synthesis. This distinction aligns with the networks’ complementary roles: the generator performs complex transformations while the discriminator implements analytical decomposition.

The optical field propagation through the generator network is visualized in Figure 2c, showing both amplitude and phase distributions at each layer for a representative input generating the digit “3”. The input Gaussian beam undergoes progressive modulation, with early layers (1–2) performing coarse shaping and later layers (3–5) refining fine details. Layer 1 immediately disperses the Gaussian profile into multiple focal points, establishing the basic digit structure. Subsequent layers redistribute energy to form the characteristic curves of the numeral, with the final layer performing precise correction to achieve uniform brightness.

Figure 2d illustrates the discriminator’s processing of a generated “fake” digit. The amplitude evolution shows progressive feature extraction, with early layers performing edge detection and spatial frequency analysis. The concentrated intensity patterns in layers 3–4 suggest the formation of characteristic features used for discrimination. The phase distributions maintain relative coherence throughout propagation, indicating that the discriminator preserves phase relationships for accurate classification. The final output layer shows clear intensity concentration in the “False” region, demonstrating successful discrimination.

The loss evolution curves in Figure 3a reveal characteristic GAN training behavior, with both networks exhibiting rapid initial improvement followed by stable oscillation around equilibrium values. The generator loss stabilizes at nearly 1.7 after approximately 100 epochs, while the discriminator loss converges to 0.69. These near-identical loss values indicate balanced competition between the networks, neither able to consistently outperform the other.

The discriminator accuracy evolution in Figure 3b provides crucial insight into the quality of generated samples. Starting from near-perfect accuracy (>95%), the discriminator’s performance gradually degrades as the generator improves, ultimately stabilizing around 50% accuracy—the theoretical Nash equilibrium for perfectly matched adversaries. At around 425 epochs, the discriminator suddenly suppressed the generator. Subsequently, as the network was trained, it returned to the equilibrium point and continued to oscillate. This demonstrated the network’s robust adversarial equilibrium, effectively resisting mode collapse or discriminator overload. This convergence to random-chance discrimination definitively indicates that the generated digits have become statistically indistinguishable from real MNIST samples.

Figure 3c–e showcases the evolution of generated digit quality across training epochs. Before epoch 100, the generated “3” digits exhibit basic structure but lack fine details. The discriminator easily identifies these as fake, evidenced by strong activation in the “False” output region. By epoch 150, significant improvement is observed: the digits display clearer definition, consistent curvature and proper proportions. The discriminator’s output becomes more ambiguous. At convergence (after epoch 150), the discriminator outputs show nearly equal intensity in both regions, confirming its inability to reliably distinguish generated samples from real samples. Additionally, we analyzed the adversarial generation of other digits (see Supplementary Material Section S3).

To evaluate the quality of generated images, we employed the SSIM and FID metrics. The average SSIM value across 1000 generated images reached 0.9573, while the FID index stood at 106.31. The reasons for the high FID index are multifaceted. Primarily, the number of images used for evaluation significantly impacts FID assessment. It should be noted that under the current settings (i.e., only 1000 evaluation images and single-class generation), the reported FID should be understood as a relative metric for comparison within the same framework, rather than an absolute benchmark directly comparable to reports from large-scale multi-class GANs. This is because FID estimates from limited samples may be biased, particularly when estimating covariance statistics from such small datasets. Typically, incorporating 10,000 or more images (more than 50,000) per calculation is required to achieve a lower index result. And we only had 1000 pictures to participate in the evaluation. Additionally, FID is influenced by the diversity of generated categories. Since we only performed generative adversarial training on the digit “3,” this inevitably contributes to the elevated FID value. Therefore, considering the SSIM index as a reference, we attribute the high FID to the numerical relationship limitations of the calculation method rather than poor image quality. More detailed information and computational methods can be found in Supplementary Material Section S1.

The diversity of generated samples, shown across different input beam configurations, demonstrates that our optical GAN successfully captures the multi-modal nature of the MNIST distribution rather than converging to a single mode—a common failure mode in GAN training. Each input beam configuration (varying position and waist size) produces distinct stylistic variations while maintaining digit recognizability, analogous to different handwriting styles in the original dataset.

3.3. D²NN-GAN for the KTH Dataset

To evaluate the scalability and generalization capability of the D²NN-GAN architecture, we extended it to the substantially more challenging KTH human action dataset [47]. This dataset presents several complications: higher resolution (160 × 120 pixels versus 28 × 28 for MNIST), complex articulated motion patterns and significant intra-class variation. We focused on the “running” action category, centering the extracted human silhouettes to ensure consistent spatial alignment.

Figure 4a,b presents the network configurations for the KTH dataset processing. The optical field evolution for KTH generation (Figure 4c–e) reveals the increased complexity of this task. The generator must synthesize coherent body poses with proper limb articulation and realistic proportions. The evolution of the light field generated by KTH (see Figure 4c) reveals the increased complexity of this task. Among them, the generator must synthesize coherent body postures with appropriate limb joints and true proportions. Figure 4d,e, respectively, show the processing of false images and real images. Among them, the most obvious difference is that during input, the intensity distribution of false images is significantly different from that of real images—false images are more concentrated in the central area of the image, while the intensity at the edges is weaker. There is no significant difference in intensity in real images. This phenomenon is mainly due to the fact that the false images are just generated by the generation network, and the limitation of modulation capability makes it impossible to balance the overall shape and intensity uniformity of the output. Therefore, it is also easier for the discriminator to identify the difference between the two in terms of intensity.

Figure 5a,b, respectively, show the loss decay curve and discriminator accuracy curve of the KTH dataset during training. Compared to MNIST training, the discriminator accuracy rapidly drops to around 20% during the steep loss decay phase. At approximately 50 epochs, the discriminator accuracy recovers to 50%. At final convergence, the discriminator accuracy stabilizes at approximately 75%. This demonstrates that the discriminator network outperforms the generator network in the KTH task.

To investigate factors limiting generation quality in the KTH dataset, we explored various input beam configurations and network architectures. Initially, we attempted to enhance generation quality by diversifying input beam modes, including higher-order Laguerre–Gaussian (LG) modes and vortex beams carrying orbital angular momentum (OAM). These alternative input configurations were hypothesized to provide richer encoding spaces. However, simulation results indicate that multiple Gaussian input modes did not significantly improve generation performance on the KTH dataset. Evaluation metrics (SSIM and FID) reveal negligible differences between outputs generated by the fundamental Gaussian mode and higher-order optical modes, suggesting that beam input diversity alone is insufficient to overcome generation challenges posed by complex motion patterns.

Following these initial investigations, we systematically explored the impact of network scale on generation quality by comparing architectures with different neuron densities per diffractive layer. Specifically, we evaluated two network configurations: 200 × 200 neurons per layer (40,000 trainable parameters per layer) and 400 × 400 neurons per layer (160,000 trainable parameters per layer). This four-fold increase in network capacity provides substantially greater representational power for learning complex optical transformations.

The generated human motion images reveal significant performance differences between network scales (see Figure 5c,d). Among these, the 200 × 200 network generated image samples exhibited significant ghosting artifacts, with SSIM indices hovering around 0.3. In contrast, the 400 × 400 network demonstrated substantial improvements, producing samples with more coherent body structures, better-defined limbs and more realistic proportions, while elevating the SSIM index to approximately 0.7. Simultaneously, the FID score for the 100 generated images decreased from 253.71 to 204.81. While still imperfect, these larger network structures captured recognizable running poses and established appropriate spatial relationships between body parts. The discriminator accuracy saturating at ~75% indicates a persistent advantage of the discriminator over the generator in this high-dimensional setting. This is because both the discriminator and generator utilize the same parameterized network architecture; generating a 160 × 120 image is significantly more challenging than inferring its authenticity. In practice, GAN stability can be improved by explicitly rebalancing generator–discriminator strength, for example, by adopting asymmetric architectures (e.g., using a larger-capacity generator while keeping a lighter discriminator) and/or by weakening discriminator updates. Therefore, for more complex tasks, matching smaller discriminator networks with larger generator networks should be considered to achieve better D²NN-GAN performance. Such asymmetric G–D configurations are particularly relevant to KTH datasets because synthesizing structured human silhouette generation is inherently more demanding than binary authenticity inference.

4. Experimental Error Discussion

Although the present work verifies D²NN-GAN in simulation, several engineering factors are critical for physical deployment, including interlayer alignment, environmental disturbances and fabrication tolerances of large-scale diffractive elements.

4.1. Quantitative Tolerance to Inter-Layer Translational Misalignment

In practical multi-layer diffractive implementations, precise layer-to-layer alignment is a key engineering challenge, because translational misalignment introduces spatial mismatch between adjacent phase masks and effectively breaks the trained optical transformation. To quantify the sensitivity of our D²NN-GAN to such alignment errors, we performed a systematic misalignment sweep based on the trained MNIST configuration.

Specifically, we considered cases where N = 1–5 diffractive layers are misaligned (“misaligned_layers”), with a translational offset of 5% (shift_percent = 0.05) or 10% (shift_percent = 0.10) of the 200 × 200-layer, corresponding to 10 px and 20 px, respectively. For each setting, the offset was applied along either the horizontal or vertical direction (“axis”). After convergence (epoch = 500), we evaluated the generation quality using FID and SSIM.

Overall,

Table 1. Misalignment tolerance analysis of the D²NN-GAN generator (MNIST).

Misaligned Layers	Shift	Axis	Final FID	SSIM (Mean ± Std)
1	5% (10 px)	Horizontal	250.79	0.9530 ± 0.0192
	5% (10 px)	Vertical	255.58	0.9530 ± 0.0163
	10% (20 px)	Horizontal	245.38	0.9480 ± 0.0279
	10% (20 px)	Vertical	181.91	0.9505 ± 0.0208
2	5% (10 px)	Horizontal	143.66	0.9523 ± 0.0239
	5% (10 px)	Vertical	157.96	0.9540 ± 0.0150
	10% (20 px)	Horizontal	242.57	0.9504 ± 0.0212
	10% (20 px)	Vertical	233.92	0.9530 ± 0.0201
3	5% (10 px)	Horizontal	167.73	0.9551 ± 0.0179
	5% (10 px)	Vertical	157.89	0.9542 ± 0.0174
	10% (20 px)	Horizontal	215.57	0.9438 ± 0.0262
	10% (20 px)	Vertical	253.72	0.9355 ± 0.0299
4	5% (10 px)	Horizontal	221.43	0.9449 ± 0.0217
	5% (10 px)	Vertical	241.58	0.9511 ± 0.0223
	10% (20 px)	Horizontal	72.96	0.9739 ± 0.0077
	10% (20 px)	Vertical	73.97	0.9733 ± 0.0119
5	5% (10 px)	Horizontal	202.62	0.9325 ± 0.0235
	5% (10 px)	Vertical	233.88	0.9354 ± 0.0304
	10% (20 px)	Horizontal	108.33	0.9586 ± 0.0116
	10% (20 px)	Vertical	154.31	0.8963 ± 0.0514

shows that SSIM remains relatively robust under moderate misalignment in many cases (typically remaining ≥ 0.93), whereas FID is noticeably more sensitive and exhibits larger fluctuations across different misalignment configurations. In this tolerance test, FID should be interpreted as a relative indicator across misalignment conditions rather than an absolute benchmark.

4.2. Temperature/Mechanical Variations

Temperature variations have negligible effects on diffractive networks (passive systems). Temperature variations typically introduce only weak perturbations in passive diffractive networks because the optical computation is carried out by free-space propagation and static phase profiles (i.e., no active biasing is required during inference). Recent integrated diffractive neural network chips have experimentally demonstrated high-temperature robustness and stable inference in harsh environments, supporting the practical stability of passive diffractive processors [48]. Mechanical vibration mainly manifests as relative interlayer misalignment (lateral shift/axial spacing/tilt) rather than intrinsic phase noise of the passive layers.

4.3. Manufacturing Precision

Regarding manufacturing precision for large-scale diffractive elements, experimental implementation of D²NN typically uses a spatial light modulator to modulate the light source and 3D printing to fabricate metasurfaces designed by an electronic computer. Limited by the precision size of 3D printing, this fabrication method is typically only available for terahertz bands [20]. Here, the D²NN-GAN operates at the wavelength of 1550 $\mathrm{n}\mathrm{m}$, which corresponds to pixel sizes of approximately 800 $\mathrm{n}\mathrm{m}$. The diffractive layer of D²NN-GAN can be fabricated by micro/nano processing technology compatible with CMOS technology, as the current state-of-the-art e-beam lithography (EBL) technology has a fabrication resolution of only a few nanometers. Nevertheless, multilayer integration still requires careful control of overlay/alignment and interlayer spacing, which remains an important engineering topic for scalable diffractive/metasurface computing hardware.

4.4. Real-Time Optimization

Beyond static fabrication, practical “real-time” optimization can be achieved through SLM-based prototyping and calibration to compensate system imperfections [49], and/or measurement-in-the-loop in situ optimization/training methods [50] recently demonstrated for diffractive optical processors and trainable on-chip diffractive architectures with massive tunable elements.

The lack of reconfigurability is an inherent characteristic of passive diffractive hardware rather than a limitation specific to the D²NN-GAN framework. Similarly to conventional large-scale generative models, once the diffractive parameters are optimized offline and deployed, the inference/generation process can run with ultra-low incremental energy cost and massive parallelism; task switching can be realized by re-fabricating/swapping phase masks or by introducing programmable diffractive layers when on-demand reconfiguration is required.

5. Conclusions

In this work, we have numerically demonstrated the feasibility of implementing generative adversarial networks using all-optical diffractive layers. The simulation results demonstrate that a dual-network architecture composed of five layers of diffractors and discriminators can achieve dynamic generative adversarial learning in the optical domain. For the MNIST digit dataset, the simulated system reaches Nash equilibrium with the discriminator accuracy stabilizing around 50%, while the generator achieves an average SSIM of 0.9573. The extension to the KTH human action dataset, though not achieving Nash equilibrium (discriminator accuracy ~75%), demonstrates the potential scalability of this approach to more complex visual data.

However, the current network still exhibits certain limitations: constrained by the depth of modulation (five layers) and the number of modulating neurons, coupled with potentially limited input encoding diversity, the network’s adversarial generation capability is somewhat restricted. This limitation is particularly evident in the results from the KTH dataset.

Despite these limitations, optical D²NN-GAN has potential natural advantages over electronic neural networks in terms of processing speed (achieved through multidimensional multiplexing of diffractive networks) and energy efficiency (the entire network consists of passive components, resulting in significantly lower overall power requirements for image generation compared to traditional neural networks). Compared to other optical GANs, D²NN-GAN integrates both optical generation and inference models within a single architecture. Especially at the input end, we can achieve the demonstration effect by using Gaussian beams of different positions and beam waist radius; that is, no active devices are needed for encoding at the input end.

Beyond the image generation demonstrated in this work, the D²NN-GAN framework can be extended in several practical directions. First, conditional or controllable generation can be realized by multiplexing additional optical input channels (e.g., spatial/wavelength/polarization encoding) to incorporate conditions, consistent with the task categories explored in optoelectronic GAN demonstrations (image generation and conditional generation) [41]. Second, image restoration (e.g., denoising or inpainting) can be formulated by training the generator with paired corrupted/target intensity distributions. Third, for spatiotemporal data, the architecture can be combined with time-multiplexed illumination or multi-frame outputs to support motion-related synthesis, providing a physically interpretable route toward optical video-oriented generative modeling. Meanwhile, it is possible to consider matching and combining large-sized generative networks with small-sized discriminative networks to enhance their ability to handle generative adversarial situations in high-dimensional complex data. In conclusion, we believe the D²NN-GAN architecture introduces a fully optical generative adversarial model, providing a practical path for future optical modeling methods such as image generation and video synthesis.