1. Introduction
Optical Coherence Tomography (OCT) [
1] is a widely used high-resolution imaging modality in clinical ophthalmology. It supports tasks such as neovascularization prediction [
2], photoreceptor disruption detection [
3], and CNN-based biomedical segmentation [
4,
5], as well as optic nerve head analysis [
6], retinal layer extraction [
7,
8], and 3D image registration [
9]. However, OCT images are inherently degraded by multiplicative speckle noise caused by the low-coherence interference of scattered light. This noise obscures tissue boundaries and reduces the reliability of automated diagnostic algorithms.
Hardware-based strategies, including angular compounding [
10], strain compounding [
11], and spatial compounding [
12,
13], can suppress speckle noise but require specialized optical configurations that are often incompatible with commercial scanners. Software-based denoising is therefore a practical requirement for tasks such as deformable registration [
14]. Early algorithmic approaches relied on spatial filters such as the SUSAN filter [
15,
16]. Researchers subsequently developed mathematical frameworks based on sparsity [
17,
18] and spectral fitting [
19]. However, these methods rely on hand-crafted priors that often fail to distinguish speckle noise from fine tissue textures, resulting in over-smoothing of clinically significant details.
Deep learning has introduced more flexible alternatives for image restoration [
20,
21,
22,
23]. Convolutional neural networks (CNNs) employing residual learning [
24] and perceptual loss functions [
25] have established high performance benchmarks for OCT denoising. Generative Adversarial Networks (GANs) [
26,
27,
28] and unsupervised paradigms such as Noise2Noise [
29,
30] and Noise2Void [
31,
32] have also been applied to model complex noise distributions. To maintain structural integrity during denoising, Ma et al. [
33] proposed an edge-sensitive cGAN that uses externally computed edge maps as additional supervision. Wang et al. [
34] introduced Caps-cGAN, which replaces standard convolutions with capsule networks and dynamic routing to capture spatial feature relationships with fewer parameters, and further employed a semi-supervised learning strategy to leverage unlabeled data. Yu et al. [
26] designed a GAN combining multi-scale convolution and dilated convolution Res-blocks for OCT despeckling. Although these GAN-based methods have advanced the field, several limitations remain. The edge-sensitive cGAN relies on external edge detection that may fail under heavy noise. Caps-cGAN uses capsule routing to model spatial relationships but does not explicitly constrain structural consistency between the denoised output and the ground truth. GAN-ResNet employs dilated convolutions for multi-scale representation yet still operates within local receptive fields. More fundamentally, standard convolutional layers [
24,
25] are constrained by local receptive fields, which can fragment the continuous structural trajectories of retinal layers. Furthermore, unregularized adversarial training [
26,
27] can introduce periodic pseudo-textures that do not correspond to any underlying anatomy.
To address these issues, we propose a Multi-Scale Self-Attention Generative Adversarial Network with a Siamese consistency constraint (SA-Siamese-GAN) for speckle noise reduction in retinal OCT images. It is the first time that a self-attention mechanism, a Siamese consistency branch, and multi-scale feature fusion are simultaneously integrated within a single GAN framework for this task. Unlike previous GAN-based OCT denoising methods, SA-Siamese-GAN uniquely combines three mechanisms: (1) a self-attention module in the generator bottleneck that captures global inter-layer pixel dependencies to preserve retinal layer continuity during filtering, (2) a Siamese consistency branch that computes
distance between the denoised output and the ground truth in a high-dimensional latent space to penalize structural discrepancies and suppress adversarial artifacts, and (3) multi-scale feature fusion in both the generator (via skip connections) and the discriminator (via cross-scale concatenation) to preserve macro-structural boundaries and micro-textural details simultaneously. The ablation study (
Section 3.4) verifies that these three components contribute synergistically. Clinical validation on 11,103 paired OCT images demonstrates that SA-Siamese-GAN achieves superior PSNR and SSIM compared to existing methods.
2. Materials and Methods
2.1. Data Acquisition and Preprocessing
Training deep neural networks requires large datasets. Since paired OCT datasets are scarce, this study utilizes two retinal OCT datasets: the Duke University dataset [
35] and the Kermany dataset [
36]. The combined dataset includes normal fundus, CNV, DR, and ERM cases. All B-scan images were cropped and resized to
pixels. Data augmentation, including horizontal flipping and random cropping, was applied to expand the dataset to 11,103 image pairs. Target clean images were generated by multi-frame averaging. Specifically, 5 repeated frames for Dataset 1 and 10 to 15 frames for Dataset 2 were registered and averaged per location. A rigid registration algorithm was used to align frames and reduce motion artifacts. Since OCT speckle noise follows a random multiplicative process with a zero-mean characteristic in the logarithmic scale, multi-frame spatial compounding suppresses speckle fluctuations to produce clean reference images. Experiments were run on a workstation with an NVIDIA GeForce RTX 3090 GPU (24 GB VRAM) using TensorFlow 2.6. The network was trained with the Adam optimizer, a batch size of 4, and 100 epochs. Initial learning rates were
for the generator and
for the discriminator. After training, image quality metrics were computed on an independent test set to evaluate denoising performance.
2.2. Proposed Denoising Network
We propose a multi-scale generative adversarial network with a self-attention mechanism and a Siamese consistency constraint (SA-Siamese-GAN) for speckle noise reduction in OCT retinal images. The overall architecture is shown in
Figure 1.
The SA-Siamese-GAN consists of three components: a multi-scale generator, a multi-scale discriminator, and a Siamese consistency branch.
The noisy OCT image x is fed into the generator, which uses an Encoder-Decoder structure with skip connections. The input passes through a convolution with Reflection Padding, Batch Normalization (BN), and ReLU activation. Three down-sampling stages (Down-1, Down-2, Down-3) use convolutions with a stride of 2, increasing feature channels from 64 to 512. The features then enter a bottleneck layer with 16 Residual Blocks ( convolutions with Dropout) and a Self-Attention (SA) layer.
In the decoding phase, three up-sampling stages (Up-1, Up-2, Up-3) use deconvolutions with a stride of 2 to restore spatial resolution. The up-sampled features are fused with the corresponding encoder features through skip connections (Skip(x_down2), Skip(x_down1), Skip(x0)). A final convolution with tanh activation and a residual addition produces the denoised image .
The reconstructed image is evaluated by a dual-branch mechanism. The multi-scale discriminator receives and the clean ground truth y alternately, processing them through a convolution with LeakyReLU and three multi-scale layers. The extracted features are resized, concatenated, and passed through fully connected (FC) layers with Sigmoid activation to output a real/fake probability (D score) in .
In the Siamese branch, and y are fed in parallel into two shared-weight CNNs. The distance between the encoded feature representations produces a similarity score (S sim) in . This constraint penalizes structural discrepancies, helping the generator preserve retinal layer continuity without introducing pseudo-artifacts.
The networks are trained alternately. For each batch, the discriminator and the Siamese network receive 8 weight updates, followed by a single weight update for the generator (critic_updates = 8). The Wasserstein loss is used without gradient penalty. No spectral normalization is applied. All three networks use the Adam optimizer (, , ) with a learning rate of .
2.3. Self-Attention Mechanism
Standard convolutions have localized receptive fields and cannot capture long-range dependencies across retinal layers in OCT images. When filtering speckle noise, local processing can fracture continuous tissue boundaries. To address this, a Self-Attention mechanism is integrated into the generator’s bottleneck. The module is shown in
Figure 2.
Given an input feature map from the preceding convolutional layers, it is flattened along the spatial dimensions to a matrix , where and is the number of input channels. To compute dependencies between any two spatial positions, undergoes three linear transformations parameterized by (implemented as convolutions), producing the Query (), Key (), and Value () matrices.
To reduce computation, the channel dimensions of
and
are compressed to
(set to
of the original channels), while the dimension
of
is unchanged. The inner product
captures global spatial correlations. The result is scaled by
and passed through a
function to produce an
attention weight matrix (
):
The Value matrix is multiplied by the attention weight matrix, producing the globally aggregated output .
Finally, is reshaped to . A learnable scalar (initialized to zero) scales the attention output, and a residual connection adds it to the original input . This lets the network learn to balance local convolution and global attention during training, preserving retinal layer continuity while suppressing noise.
2.4. Discriminator and Siamese Consistency Module
A multi-scale feature fusion discriminator evaluates the realism of generated images. As shown in the lower-left of
Figure 1, the discriminator consists of five convolutional blocks. The first block uses a
kernel with stride 2, same-padding, and LeakyReLU (
) without Batch Normalization. The subsequent three blocks each use
kernels with stride 2, same-padding, LeakyReLU, and Batch Normalization. The fifth block uses a
kernel with stride 1. The channel dimensions are 64, 128, 256, 512, and 512, respectively. For multi-scale fusion, feature maps from all five blocks are bilinearly resized to the spatial resolution of the first block and concatenated. A
convolution is then applied for feature integration, followed by Batch Normalization and LeakyReLU. The output is flattened and passed through two fully connected layers (1024 with tanh, then 1 with sigmoid) to produce the final probability (
D score) in
.
Binary discriminator feedback alone is often insufficient to maintain anatomical fidelity, and can lead to texture distortion during adversarial training. To address this, a Siamese network is introduced as a consistency module, as shown in the lower-right of
Figure 1. This branch consists of two parallel CNNs with shared weights. During training,
and
y are fed into the two branches. The shared CNN maps each image to a high-dimensional feature space, and the
distance between the two feature representations is mapped to a similarity score (
S sim) in
. This Siamese constraint penalizes structural differences, guiding the generator to preserve retinal layer continuity without introducing pseudo-artifacts.
2.5. Joint Loss Functions
Optimizing a generative model with a single adversarial objective often involves a trade-off between speckle suppression and the preservation of anatomical structures. The optimization objective of SA-Siamese-GAN combines three loss functions with the similarity feedback from the Siamese branch. The total loss
is:
where
, and
are hyperparameter weights. In our experiments, these were set to
,
,
, and
.
is the Wasserstein adversarial loss. Unlike the Jensen-Shannon divergence used in standard GANs, the Earth-Mover distance optimized by
reduces mode collapse and stabilizes training:
where
and
denote the real ground truth distribution and the generated data distribution, respectively.
is the perceptual loss. Instead of pixel-wise error, it compares high-level semantic features extracted by a pre-trained VGG-16 network (the
block3_conv3 layer) from
and
y:
where
denotes the semantic feature maps extracted by the specified layer of the VGG network, and
N represents the total number of elements in the feature maps.
is the Structural Similarity loss, minimized as the negative SSIM index to constrain luminance, contrast, and structure:
is the structural discrepancy penalty from the Siamese branch. The shared CNN encodes
y and
into latent vectors. Their
distance is passed through a fully connected layer with Sigmoid activation, producing a similarity score
. The loss then penalizes deviations from the expected target (
during generator optimization):
3. Results
3.1. Visual Evaluation of the Proposed Model
Subjective visual evaluation is important for assessing the clinical utility of denoising methods.
Figure 3 shows the denoising results of our model on representative OCT retinal slices.
As shown in
Figure 3, the original images contain speckle noise that obscures boundaries between retinal layers. The denoised outputs (b, d, f, h) remove most background speckle noise. Even under heavy noise (e.g.,
Figure 3a,e,g), the model produces smooth backgrounds while retaining the continuous layered structure of the retina. The network avoids the over-smoothing common in traditional filters.
To further illustrate the micro-structural preservation capabilities of the proposed architecture,
Figure 4 provides a magnified view of the Region of Interest (ROI) marked by the red bounding box in
Figure 3e.
The magnified ROI shows a morphologically complex region. While speckle masks edge gradients in the original noisy crop, the contours and textural variations are reconstructed in the denoised image. This preservation of fine structure results from the self-attention mechanism and Siamese constraints, which penalize structural distortions.
To assess generalization across different pathological conditions, we evaluated SA-Siamese-GAN on three representative retinal cases: choroidal neovascularization (CNV), diabetic macular edema (DME), and a normal retina. As shown in
Figure 5, the model effectively suppresses speckle noise in all three conditions. In the CNV case, the sub-retinal fluid boundary and neovascular membrane remain clearly visible. In the DME case, intra-retinal cystoid spaces and the distorted layer architecture are well preserved. In the normal case, the intact layered structure of the retina is maintained without over-smoothing. These results demonstrate that SA-Siamese-GAN is not specialized to a single disease morphology and generalizes across diverse tissue structures.
3.2. Qualitative Comparison with State-of-the-Art Methods
We compared SA-Siamese-GAN against six denoising methods: Wavelet, BM3D, SBSDI, NLM-DIP, MNLM-DIP, and GAN-ResNet.
Figure 6 shows the denoising results on a heavily corrupted OCT slice, and
Figure 7 provides a magnified view of the central foveal region.
As shown in
Figure 6a, the original OCT image is contaminated by dense speckle noise, particularly in the apical and background regions, which impairs diagnostic visibility. The Wavelet method (
Figure 6b) removes high-frequency noise but introduces block-like artifacts that distort the tissue appearance. BM3D (
Figure 6c) reduces noise but leaves residual speckle and causes slight structural distortions. SBSDI (
Figure 6d) over-smoothes the image, blurring retinal layer boundaries.
The deep learning models NLM-DIP (
Figure 6e) and MNLM-DIP (
Figure 6f) balance noise suppression and structure retention better, but still do not fully recover the hierarchical textures of retinal layers. GAN-ResNet (
Figure 6g) produces relatively sharp boundaries. However, constrained by its localized convolutions and unregularized adversarial training, it produces periodic pseudo-textures, including false anatomical lines in the background.
The magnified crops in
Figure 7 confirm these observations. SA-Siamese-GAN (
Figure 7h) produces the best visual quality. It suppresses background speckle while reconstructing retinal layer boundaries with continuous clarity, and preserves hierarchical textural details without synthetic artifacts.
3.3. Quantitative Evaluation
To supplement visual assessment, we evaluated PSNR and SSIM across two test datasets: OCT-1 (Duke dataset [
35]) and OCT-2 (Kermany dataset [
36]). Results are shown in
Table 1.
As shown in
Table 1, SA-Siamese-GAN outperforms all other methods on both datasets.
Traditional filtering methods (Wavelet, BM3D) produce the lowest PSNR and SSIM scores (e.g., BM3D achieves an SSIM of 0.56 on OCT-1), consistent with the blurring and block artifacts observed in the visual comparison. The deep-learning DIP variants (NLM-DIP, MNLM-DIP) improve PSNR to 27–29 dB, but their SSIM scores suggest that high-frequency structural details remain compromised.
GAN-ResNet achieves a PSNR of 29.85 dB and an SSIM of 0.88 on OCT-1. SA-Siamese-GAN improves PSNR to 34.52 dB on OCT-1 (a gain of 4.67 dB) and 33.13 dB on OCT-2, with SSIM reaching 0.91 on both datasets. These results indicate that the self-attention mechanism and Siamese constraints together enable better speckle removal while preserving anatomical structure. Notably, the performance gap between the two datasets is small (1.39 dB in PSNR and identical SSIM of 0.91), despite their different acquisition protocols and device configurations. This consistency provides evidence of cross-domain robustness. The hyperparameters of all baseline methods are listed in
Appendix A.
3.4. Ablation Study
To isolate the contribution of each architectural component, ablation experiments were conducted under a unified reduced configuration ( resolution, 6 residual blocks, 6000 training pairs, and 20 training epochs) to manage computational cost. All variants shared the same training data, random seed, and optimizer settings, with PSNR and SSIM evaluated on a held-out test set of 1000 images. Four variants were compared: (1) the full SA-Siamese-GAN, (2) without the self-attention module in the generator bottleneck (w/o SA), (3) without the Siamese consistency branch (w/o Siamese), and (4) without multi-scale feature fusion in the discriminator (w/o MS-D).
As shown in
Table 2, the full model achieves the highest PSNR (37.21 dB) and SSIM (0.9643). Removing any single module consistently degrades both metrics, confirming that each component contributes positively to denoising performance. The multi-scale discriminator contributes the largest individual gain (+0.45 dB). Without multi-scale fusion, the discriminator relies solely on the deepest feature map, which weakens its capacity to capture fine-grained texture discrepancies and reduces the quality of adversarial gradients provided to the generator.
The self-attention module contributes +0.26 dB. While this numerical gain appears modest, pixel-wise metrics alone may understate its value: the self-attention mechanism captures long-range dependencies across the entire feature map, which is essential for preserving the continuity of retinal layer boundaries. As shown in the visual comparison (
Figure 8), the w/o SA variant exhibits subtle layer fragmentation that is not fully reflected in the aggregate PSNR score. Similarly, the Siamese branch contributes +0.21 dB. Its primary role is to regularize the generator against pseudo-artifacts during adversarial training—a qualitative benefit visible in the background regions of the denoised outputs.
The combined gain of all three modules (+0.45 dB) is achieved through their synergistic interaction: the multi-scale discriminator enforces global texture realism, the self-attention mechanism preserves structural continuity, and the Siamese branch suppresses spurious anatomical patterns. Removing any one component weakens this synergy, demonstrating that the integrated design of SA-Siamese-GAN is necessary for optimal speckle noise reduction in retinal OCT images.
At the standard configuration (512 × 512, 16 residual blocks), average inference time per B-scan is 26.8 ms on an RTX 3090 GPU (∼37 FPS). Reducing the number of residual blocks to 6 lowers this to 17.7 ms, offering a lightweight variant with a minor trade-off in denoising quality. The complexity of the self-attention module is O(H2W2), quadratic in spatial resolution, resulting in 6.8 ms at 256 × 256 versus 26.8 ms at 512 × 512.
4. Discussion
In this article, we proposed SA-Siamese-GAN for speckle noise reduction in retinal OCT images. It is the first time that a self-attention mechanism is combined with a Siamese consistency branch within a multi-scale GAN framework for this task. Unlike previous GAN-based OCT denoising methods that rely solely on local convolutional operations, SA-Siamese-GAN captures global inter-layer dependencies through self-attention while the Siamese branch penalizes structural discrepancies in latent space. The ablation study (
Section 3.4) confirmed that each component contributes independently. The multi-scale discriminator provided the largest gain (+0.45 dB PSNR). The self-attention module and the Siamese branch contributed +0.26 dB and +0.21 dB, respectively. It can be observed from the visual comparison (
Figure 8) that the w/o SA variant exhibits subtle layer fragmentation, which indicates that pixel-wise metrics alone understate the structural role of self-attention.
As shown in
Table 1, SA-Siamese-GAN achieved a PSNR of 34.52 dB and an SSIM of 0.91 on OCT-1, outperforming all compared methods. Traditional methods such as Wavelet and BM3D produced block-like artifacts and excessive blurring. BM3D achieved an SSIM of only 0.56 on OCT-1. These results confirm that local mathematical priors are insufficient for OCT speckle noise. Deep-learning variants such as NLM-DIP improved PSNR to the 27–29 dB range, but high-frequency textures remained compromised, which may be caused by their restricted receptive fields. GAN-ResNet generated periodic pseudo-textures in background regions. This points to a known limitation of standard GAN architectures: without structural supervision, the generator can produce textures that fool the discriminator without being anatomically correct.
The self-attention module introduces computational overhead that scales quadratically with spatial resolution. As reported in
Section 3.4, inference at 512 × 512 takes 26.8 ms per B-scan on an RTX 3090 GPU. Reducing residual blocks from 16 to 6 lowers this to 17.7 ms. This latency remains well within clinical requirements. Future work could explore linear-complexity attention variants to further reduce computational cost.
There are still several limitations in this study. The comparison methods do not include recent Transformer- or diffusion-based OCT denoising models (e.g., THFN-OCT [
37], GARD [
38]), because their code and pretrained weights are not publicly available as of this writing. The method has only been validated on 2D B-scans. Although cross-pathology generalization has been demonstrated qualitatively (
Figure 5) and the two datasets originate from different devices, a systematic evaluation under varying scan densities and across additional commercial OCT platforms requires dedicated multi-manufacturer data collection and is planned for future work. Extension to 3D volumetric OCT data and real-time 4D-OCT denoising requires further investigation. The benefit of improved denoising on downstream tasks such as retinal layer segmentation and lesion detection has not been quantified. Therefore, evaluating the impact of SA-Siamese-GAN on these clinical tasks is one of our future research directions. Besides, how to further reduce the parameter count while maintaining denoising performance is another focus that will be continuously explored in future work.