A Dual-Generator Translation Network Fusing Texture and Structure Features for SAR and Optical Image Matching

Abstract

The matching problem for heterologous remote sensing images can be simplified to the matching problem for pseudo homologous remote sensing images via image translation to improve the matching performance. Among such applications, the translation of synthetic aperture radar (SAR) and optical images is the current focus of research. However, the existing methods for SAR-to-optical translation have two main drawbacks. First, single generators usually sacrifice either structure or texture features to balance the model performance and complexity, which often results in textural or structural distortion; second, due to large nonlinear radiation distortions (NRDs) in SAR images, there are still visual differences between the pseudo-optical images generated by current generative adversarial networks (GANs) and real optical images. Therefore, we propose a dual-generator translation network for fusing structure and texture features. On the one hand, the proposed network has dual generators, a texture generator, and a structure generator, with good cross-coupling to obtain high-accuracy structure and texture features; on the other hand, frequency-domain and spatial-domain loss functions are introduced to reduce the differences between pseudo-optical images and real optical images. Extensive quantitative and qualitative experiments show that our method achieves state-of-the-art performance on publicly available optical and SAR datasets. Our method improves the peak signal-to-noise ratio (PSNR) by 21.0%, the chromatic feature similarity (FSIMc) by 6.9%, and the structural similarity (SSIM) by 161.7% in terms of the average metric values on all test images compared with the next best results. In addition, we present a before-and-after translation comparison experiment to show that our method improves the average keypoint repeatability by approximately 111.7% and the matching accuracy by approximately 5.25%.

1. Introduction

Different sensors can capture different features. In particular, synthetic aperture radar (SAR) images and optical images are widely used in map production [1]. Optical images conform to human vision, but are susceptible to objective factors such as cloud interference [2], whereas SAR images are immune to the imaging defects of optical images and have the advantages of all-weather acquisition, a long line of sight, and some level of penetration capability. Therefore, the fusion of optical and SAR images is widely used in pattern recognition [3], change detection [4], and landslide recognition [5]. However, a prerequisite for the fusion of SAR and optical images is high-accuracy matching. In recent decades, many feature matching methods for homologous images have been proposed, e.g., SIFT [6], SURF [7], ORB [8], and LoFTR [9]. The LoFTR method mainly focuses on dense matching of weakly textured regions of homologous images. However, these methods are applicable to homologous image matching, but not to SAR and optical image matching because NRDs are not considered. Recently, to address severe NRDs between SAR and optical images, Cui et al. [10] implemented MAP-Net by introducing spatial pyramid aggregated pooling (SPAP) and an attention mechanism to improve the matching precision of optical and SAR images. Li et al. [11] proposed the radiation-variation insensitive feature transform (RIFT) for different types of images. Cui et al. [12] extended scale invariance based on RIFT, but their method was more sensitive to noise, and Li et al. [13] proposed the locally normalized image feature transform (LNIFT) using a local normalization filter to convert images of different modalities into the same intermediate modality, turning the multimodal image matching problem into a homogenous matching problem, and making different modalities similar to improve the matching accuracy. In recent years, deep learning has been successfully introduced into the field of remote sensing image processing for applications such as image matching [14], image fusion [15], and image translation [16]. It is noteworthy that generative adversarial networks (GANs) can better convert multimodal image matching problems into homologous matching problems. Many researchers have implemented matching between SAR and optical images based on SAR-to-optical translation. Quan, D. [17] proposed a generative matching network (GMN) to generate a corresponding simulated optical image for a real SAR image or a pseudo-SAR image for a single optical image, and then input these matched pairs into a matching network to infer whether they matched, achieving improved performance in SAR–optical image matching. Merkle et al. [18] jointly implemented the translation of single-polarization SAR images into optical images by means of a conditional generative adversarial network (CGAN) and verified the possibility of using the transformed pseudo-optical images for image matching. A k-means clustering-guided generative adversarial network (KCG-GAN) [19] has also been proposed for use in SAR and optical image matching, and the results showed that the quality of SAR-to-optical translation limits the matching accuracy between SAR and optical images. Therefore, the key question that needs to be urgently answered is how to design a high-precision SAR-to-optical translation method to enhance the SAR–optical matching performance.

In recent decades, many researchers have proposed methods, which are mainly based on image enhancement algorithms and pseudo-colour encoding algorithms, for SAR–optical translation. In the field of image enhancement, a wavelet transform-based method was used for SAR image denoising to achieve SAR image enhancement from the perspective of noise suppression, but it was found that there was a possibility of increasing the amount of other types of clutter [20]. By introducing visualization algorithms to map high-dynamic-range SAR amplitude values to low-dynamic-range displays via reflectivity distortion, entropy maximization can be preserved to improve the visual quality of SAR images by maximizing the display information [21], and an adaptive two-scale enhancement method can be used to visualize all greyscale information and enhance local target peaks [22]. However, the previous approaches enhance SAR images by means of visualization methods that cannot effectively resolve differences caused by nonlinear radiation distortion (NRDs). In the field of pseudo-colour coding, the pixels of SAR images are mainly encoded to make them as similar as possible to those of optical images [23,24,25]; however, a greyscale image is obtained instead of a three-channel image, and because the results are highly dependent on the specifics of the model, the performance may decline in practical use. The images processed by image enhancement algorithms and pseudo colour encoding algorithms are enhanced in terms of visual features, but both types of algorithms ignore the NRDs differences between SAR images and optical images; consequently, large differences in structure and texture remain in the resulting pseudo-optical images compared to the real optical images. For the task of automatic image colorization, a deep learning model can be used to predict the pixel-by-pixel colour histogram suitable for the colouring task without structurally transformed image pairs [26]. In the field of SAR image processing, a convolutional neural network (CNN)-based approach has been used to convert a single-polarization greyscale SAR image into a full-polarization image [27]. Moreover, generative adversarial networks (GANS) [28] are widely used for image translation. A dialectical GAN using conditional Wasserstein generative adversarial network–gradient penalty (WGAN-GP) loss functions has been applied to translate Sentinel-1 images into TerraSAR-X images [29]. Based on the proposal of a boundary equilibrium generative adversarial network (BEGAN) [30], an adversarial network was designed for SAR image generation, and it was demonstrated that the proposed method could improve the classification accuracy [31]. Many GAN-based methods have also been used in SAR-to-optical transformation, such as Pix2pix [32], CycleGAN [33], S-CycleGAN [34], and EPCGAN [16]. Pix2pix and CycleGAN can both be used for SAR–optical translation, but they have certain drawbacks. With Pix2pix, the structure is vague, and some objects have missing structural information, whereas CycleGAN can retain structural information, but ignores land cover information; accordingly, S-CycleGAN combines the advantages of CycleGAN, preserving both land cover information and structural information. He, W. [35] proposed a model combining residual networks and CGANs that can simulate optical images from multitemporal SAR images. However, there is a major problem with such methods; they usually rely on a network structure designed for optical image transformation, with only simple modifications, which is not applicable for SAR–optical translation because of the differences between the imaging principles of SAR images and optical images. Based on this understanding, a feature-guided method based on a discrete cosine transform (DCT) loss has been proposed [36], and edge information has been used to guide SAR–optical translation to obtain pseudo-optical images with better edge information [37]. Similarly, EPCGAN considers the edge blurring problem for pseudo-optical images, and uses gradient information to preserve the edge information in generated pseudo-optical images. The pseudo-optical images obtained in this way contain better structural information, but a situation may arise in which structure and texture features cannot be effectively fed back, resulting in poor and unrealistic imaging effects due to the inability to achieve deep fusion of the structure and texture features. Inspired by [38], in which more natural image inpainting results were obtained by means of a two-branch network, we also treat SAR-to-optical image translation as consisting of two complementary subtasks, namely, texture translation and structure translation, considering the NRDs of SAR images. We reduce the gap between pseudo-optical and real optical images by introducing a spatial-domain loss function and frequency-domain loss function, and thus, obtain pseudo-optical translation results with high accuracy (see Figure 1).

In this paper, we propose a dual-generator translation network that fuses texture and structure features to obtain enhanced pseudo-optical images for SAR–optical matching. The proposed network consists of dual generators, bidirectional gated feature fusion (Bi-GFF) [38], and contextual feature aggregation (CFA) [39] modules and discriminators. First, the input SAR image is decomposed into structure and texture features based on the Canny edge detection algorithm [40]. Then, the structure features and greyscale map are input into the structure encoder, the SAR image is input into the texture encoder, and the feature maps of different dimensions from the texture encoder and structure encoder are stitched together to join the structure and texture decoder to obtain texture features and structure features, which are then fused and refined by the Bi-GFF module and CFA module. Finally, a frequency-domain loss function (focal frequency loss [41]) and a spatial-domain loss function (mean square error) are introduced to reduce the gap between the pseudo-optical and real optical images during the learning process. We present comparative experiments and ablation experiments conducted on the same dataset. The experimental results show that the proposed method yields images with clearer textures and structures that are used to achieve better evaluation results that exhibit better visual properties than the results of Pix2pix [32], CycleGAN [33], S-CycleGAN [34], and EPCGAN [16].

Specifically, the major contributions of this paper are as follows:

• We propose a dual-generator translation network that fuses texture and structure features to improve the matching of SAR images with optical images. The proposed network includes both structure and texture generators, and the structure and texture features are coupled with each other by these dual generators to obtain high-quality pseudo-optical images.
• We introduce spatial-domain and frequency-domain loss functions to reduce the gap between pseudo-optical images and real optical images, and present ablation experiments to prove the superiority of our approach.
• To demonstrate the superiority of the proposed algorithm, we select training and test data from public datasets, and we present keypoint detection and matching experiments for comparisons between pseudo-optical images and real optical images and between real optical images and SAR images before and after translation.

The remainder of this paper is organized as follows. The proposed dual-generator translation network fusing texture and structure features for SAR–optical image translation is introduced in Section 2. We present the experimental results and matching applications in Section 3. A discussion is provided in Section 4. Finally, the conclusions are summarized in Section 5.

2. Methods

In this section, we introduce the proposed dual-generator translation network fusing texture and structure features for SAR–optical image matching. As illustrated in Figure 2, the dual generators provide feedback to each other to obtain the structure and texture features, which are fused by the Bi-GFF and CFA modules. In the following subsections, we present the details of the generators, discriminator, and loss functions.

2.1. Generators

As shown in Figure 2, the generator part of the SAR-to-optical translation network is divided into two generators, namely, a structure generator and a texture generator, which are based on U-Net variants [42], where final features from the structure encoder and multilevel features from the texture encoder are added to the texture decoder via a skip connection, and final features from the texture encoder and multilevel features from the structure encoder are added to the structure decoder via a skip connection. We also show the structural details of the texture and structure generators in

Table 1. Details of the texture and structure generator architecture. PConv is defined as a partial convolutional layer with the specified filter size, stride, and padding. Concat indicates that structure features and texture features are connected by a skip connection.

Module Name	Filter Size	Channel	Stride	Padding	Nonlinearity
Texture/Structure (T/S) Encoder
T/S Input		3/2
T/S Encoder PConv1	7 × 7	64	2	3	ReLU
T/S Encoder PConv2	5 × 5	128	2	2	ReLU
T/S Encoder PConv3	5 × 5	256	2	2	ReLU
T/S Encoder PConv4	3 × 3	512	2	1	ReLU
T/S Encoder PConv5	3 × 3	512	2	1	ReLU
T/S Encoder PConv6	3 × 3	512	2	1	ReLU
T/S Encoder PConv7	3 × 3	512	2	1	ReLU
Texture Decoder
S Encoder-PConv7		512	-	-	-
Concat (S Encoder-PConv7, T Encoder-PConv6)		512 + 512	-	-	-
T Decoder PConv8	3 × 3	512	1	1	LeakyReLU
Concat (T Decoder PConv8, T Encoder-PConv5)		512 + 512	-	-	-
T Decoder PConv9	3 × 3	512	1	1	LeakyReLU
Concat (T Decoder PConv9, T Encoder-PConv4)		512 + 512	-	-	-
T Decoder PConv10	3 × 3	512	1	1	LeakyReLU
Concat (T Decoder PConv10, T Encoder-PConv3)		512 + 256	-	-	-
T Decoder PConv11	3 × 3	256	1	1	LeakyReLU
Concat (T Decoder PConv11, T Encoder-PConv2)		256 + 128	-	-	-
T Decoder PConv12	3 × 3	128	1	1	LeakyReLU
Concat (T Decoder PConv12, T Encoder-PConv1)		128 + 64	-	-	-
T Decoder PConv13	3 × 3	64	1	1	LeakyReLU
Concat (T Decoder PConv13, T Input)		64 + 3	-	-	-
Texture Feature	3 × 3	64	1	1	LeakyReLU
Structure Decoder
T Encoder-PConv7		512	-	-	-
Concat (T Encoder-PConv7, S Encoder-PConv6)		512 + 512	-	-	-
S Decoder PConv14	3 × 3	512	1	1	LeakyReLU
Concat (S Decoder PConv14, T Encoder-PConv5)		512 + 512	-	-	-
S Decoder PConv15	3 × 3	512	1	1	LeakyReLU
Concat (S Decoder PConv15, T Encoder-PConv4)		512 + 512	-	-	-
S Decoder PConv16	3 × 3	512	1	1	LeakyReLU
Concat (S Decoder PConv16, T Encoder-PConv3)		512 + 256	-	-	-
S Decoder PConv17	3 × 3	256	1	1	LeakyReLU
Concat (S Decoder PConv17, T Encoder-PConv2)		256 + 128	-	-	-
S Decoder PConv18	3 × 3	128	1	1	LeakyReLU
Concat (S Decoder PConv18, T Encoder-PConv1)		128 + 64	-	-	-
S Decoder PConv19	3 × 3	64	1	1	LeakyReLU
Concat (S Decoder PConv19, S Input)		64 + 2	-	-	-
Structure Feature	3 × 3	64	1	1	LeakyReLU

. In the encoder stage, the SAR image to be translated is passed to the texture encoder, and the greyscale image and edge structure image of the SAR image to be translated are passed to the structure encoder. In the decoder stage, the structure features from the structure encoder are used as constraints in the texture decoder, and the texture features from the texture encoder are used as constraints in the structure decoder. This coupled dual-generator structure ensures good complementarity between the structure and texture features. Compared with normal convolutional layers, partial convolutional layers can better capture the information of irregular boundaries [42]; accordingly, considering the severe scattering noise, NRD, and large irradiance differences between optical and SAR images, we also use partial convolutional layers instead of normal convolutional layers. In addition, we add skip connections in the CFA module to join together low-level and high-level features during the fusion of the structure and texture features to ensure robust prediction results.

After the texture and structure generators have obtained their respective features, the Bi-GFF module is applied to fuse the structure and texture features to enhance their consistency, and then, the CFA module is applied to further refine the generated pseudo-optical image.

Bidirectional Gated Feature Fusion (Bi-GFF): This module follows the structure and texture generators, and implements information exchange between the structure and texture features, as shown in Figure 3. The texture features are denoted by $f_t$, the structure features are denoted by $f_s$, and the features after information exchange can be expressed as:

$$\acute{f_s}=f_s\oplus \left(W_s\left(Concat\left(f_t,f_s\right)\right)\otimes f_t\right)$$

(1)

$$\acute{f_t}=f_t\oplus \left(W_t\left(Concat\left(f_t,f_s\right)\right)\otimes f_s\right)$$

(2)

where $\oplus$ denotes elementwise addition, $\otimes$ denotes elementwise multiplication, and $W_s$ and $W_t$ denote the convolutional layer mapping functions with a convolutional kernel of 3.

Finally, $\acute{f_s}$ and $\acute{f_t}$ are fused at the channel level to obtain the fused features:

$$f=Concat\left(\acute{f_s},\acute{f_t}\right)$$

(3)

Contextual Feature Aggregation (CFA): As shown in Figure 4, the CFA module is introduced to determine which information in the SAR image contributes to SAR-to-optical translation, thereby enhancing the correlation between image features, and ensuring the overall consistency of the image.

First, the feature map F is divided into ${\displaystyle \sum }_{i=1}^n{F}_{x_i}$ 3 × 3 patches after being encoded in convolutional layers, and attention scores are obtained by a contextual attention layer, which calculates the cosine similarity between each pair of patches and applies the softmax function to this similarity to obtain the corresponding attention score. Then, the attention scores are multiplied by the 3 × 3 patches to obtain a reconstructed feature map. The contextual attention layer is defined as:

$$F=EncoderConv\left(F\right)$$

(4)

$$s_{Attention}=\frac{exp(\langle \frac{F_{x_i}}{\Vert F_{x_i}\Vert {}_2},\frac{F_{x_j}}{\Vert F_{x_j}\Vert {}_2}\rangle )}{{{\displaystyle \sum }}_{i=1}^{n}exp(\langle \frac{F_{x_i}}{\Vert F_{x_i}\Vert {}_2},\frac{F_{x_j}}{\Vert F_{x_j}\Vert {}_2}\rangle )}$$

(5)

$$F_{rec}={{\displaystyle \sum }}_{i=1}^n{F}_{x_i}*s_{Attention}$$

(6)

where $F_{x_i}$ and $F_{x_j}$ denote the i-th and j-th patches, respectively; $s_{Attention}$ denotes the attention scores; and $F_{rec}$ denotes the reconstructed feature map.

Then, multiscale semantic features are captured from the reconstructed feature map by using different dilation rates:

$$f_{rec}^k=Con{v}_k\left(F_{rec}\right)$$

(7)

where $Con{v}_k\left(.\right)$ denotes the k-th dilated convolution layer, $k\in \left\{1,2,4,8\right\}$.

A weight generator module is defined to produce pixel-level prediction maps, which are split into four weight modules:

$$W^1,W^2,W^4,W^8=Slice\left(W\right)$$

(8)

$$F_{rec}=(F_{rec}^1\otimes W^1)\oplus (F_{rec}^2\otimes W^2)\oplus (F_{rec}^4\otimes W^4)\oplus (F_{rec}^8\otimes W^8)$$

(9)

Finally, we use skip connections to splice F and f1 to prevent semantic information from being lost.

$$\acute{F}=DecoderConv(Concat(F_{rec},F))$$

(10)

2.2. Discriminator

The discriminator [38] distinguishes pseudo-optical images from real optical images by means of two branches: a texture branch and a structure branch. The last layer of the discriminator uses the sigmoid nonlinear activation function, and the structure branch has the same architecture as the texture branch. In the structure branch, the mapping for edge detection is first obtained by a residual network module [43] and a convolutional layer with a kernel size of 1. Then, the structure features are obtained by splicing with greyscale features. In the texture branch, the pseudo-optical image is directly mapped to obtain texture features, and finally, it is stitched to compute the adversarial loss. In addition, we apply spectral normalization [44] in the network to effectively solve the instability problem during network training.

2.3. Loss Functions

The algorithm proposed in this paper includes two generators, a texture generator $G_t$ and a structure generator $G_s$, and a discriminator to learn the translation from the SAR image domain ${\left\{x_i\right\}}_{i=1}^n\in X$ to the optical image domain ${\left\{y_i\right\}}_{i=1}^n\in Y$. The original SAR image $x_i^{SAR}$, the greyscale image $x_i^{gray}$ of the SAR image, and the edge image $x_i^{Edge}$ of the SAR image are passed to the generators to generate the texture features $f_t$ through the texture generator and the structure features $f_s$ through the structure generator. The texture and structure features are then fused by the Bi-GFF module ${\beta }_{Bi-GFF}$ and the CFA module ${\sigma }_{CFA}$ to obtain the pseudo-optical image $Y_{pseudo}$. The discriminator D is similarly divided into two branches, i.e., a structure branch $D_s$ and a texture branch $D_t$. The edge structure image $Y_{pseudo}^{Edge}$ obtained through edge detection convolution is input into the structure branch of the discriminator, the generated pseudo-optical image is input into the texture branch of the discriminator, and finally, the features from the two branches are concatenated in the channel dimension to distinguish a real optical image $Y_{real}$ from a generated pseudo-optical image $Y_{pseudo}$:

The generator is defined as:

$$Y_{\mathit{pseudo}}={\sigma }_{CFA}\left({\beta }_{Bi-GFF}\left(\left\{f_t,f_s=\left(G_t\left(x_i^{SAR}\right),G_s\left(x_i^{\mathrm{gray}},x_i^{\mathrm{Edge}}\right)\right)\right\}\right)\right)$$

(11)

The discriminator is defined as:

$$Real/Fake=D\left(\left\{D_t\left(Y_{pseudo}\right),D_s\left(Y_{pseudo}^{Edge},Y_{pseudo}^{Gray}\right)\right\}\right)$$

(12)

where $\left(.\right)$ denotes the projection function implemented by the convolutional layer and $\left\{.\right\}$ denotes concatenation in the channel dimension.

Reconstruction Loss: We define the reconstruction loss in terms of the differences between a real optical image and the corresponding pseudo-optical image obtained after the Bi-GFF and CFA modules have fused the structure and texture features.

(1) The mean square error (MSE) loss function is adopted to reduce the difference in the spatial domain between the pseudo-optical and real optical images at the pixel level. This loss function has the following form:

$$l_{MSE}^{rec}=\mathbb{E}\left[\Vert Y_{pseudo}-Y_{real}\Vert {}_2\right]$$

(13)

(2) The focal frequency loss (FFL) function is adopted to reduce the difference between the pseudo-optical and real optical images in the frequency domain, and to reduce the artefacts in the pseudo-optical image. The FFL function was proposed in [41]. We first use the 2D discrete Fourier transform (DTF) to separately adjust the frequency representations of the pseudo-optical image and the real optical image, dividing each frequency value by $\sqrt{HW}$ for standard orthogonalization to obtain a smooth gradient, and adjusting the spatial frequency weights of each image by means of a dynamic spectral weight matrix $w\left(u,v\right)$. Then, the FFL function can be expressed as:

$$F{(u,v)}_{\mathit{pseudo}}^Y={\sum }_{x=0}^{H-1}{\sum }_{y=0}^{W-1}f(x,y)·e^{-i2\pi {\left(\frac{ux}{H}+\frac{vy}{W}\right)}_{}}$$

(14)

$$F{(u,v)}_{\mathit{real}}^Y={\sum }_{x1=0}^{H-1}{\sum }_{y1=0}^{W-1}f\left(x_1,y_1\right)·e^{-i2\pi \left(\frac{ux}{H}+\frac{vy}{W}\right)}$$

(15)

$$l_{FFL}^{rec}=\frac{1}{HW}{\sum }_{u=0}^{H-1}{\sum }_{v=0}^{W-1}w(u,v){\left|F{(u,v)}_{\mathit{pseudo}}^Y-F{(u,v)}_{\mathit{real}}^Y\right|}^2$$

(16)

where $F{\left(u,v\right)}_{pseudo}^Y$ denotes a frequency value in the pseudo-optical image, $F{\left(u,v\right)}_{real}^Y$ denotes the corresponding frequency value in the real optical image, (u, v) represents the coordinates of a spatial frequency in the frequency spectrum, H × W denotes the size of the image, (x, y) denotes the coordinates of an image pixel in the spatial domain, f(x, y) is the corresponding pixel value, and $w\left(u,v\right)$ denotes the dynamic spectral weight matrix.

(3) The VGG loss is used for the perceptual loss of the pseudo-optical and real optical images in terms of high-level semantic information. The pseudo-optical and real optical images are input into the VGG model pretrained on ImageNet [45] to obtain their high-level semantic information. The VGG loss can then be expressed as:

$$l_{VGG}^{rec}=\mathbb{E}\left[{\displaystyle \sum }_i^3\Vert {\phi }_{VGG}^i\left(Y_{pseudo}\right)-{\phi }_{VGG}^i{\left(Y_{real}\right)\Vert }_1\right]$$

(17)

where ${\phi }_{VGG}^i$(.) denotes the projection function of the i-th pooling layer of the pretrained VGG network model.

(4) A style loss is used to ensure that SAR images are translated into pseudo-optical images with the same style as real optical images. The style loss can be expressed as:

$$l_{Style}^{rec}=\mathbb{E}\left[{\displaystyle \sum }_i^3\Vert {\mu }_{VGG}^i\left(Y_{pseudo}\right)-{\mu }_{VGG}^i{\left(Y_{real}\right)\Vert }_1\right]$$

(18)

where ${\mu }_{VGG}^i$(.) = ${\phi }_{VGG}^i{(.)}^T$, with ${\phi }_{VGG}^i(.)$ denoting the Gram matrix constructed from the activation map ${\phi }_{VGG}^i$. We choose to use the style loss [46] as demonstrated by Sajjadi et al. [47], based on its effectiveness in eliminating checkerboard artefacts.

Adversarial Loss: We define the adversarial loss in terms of a criterion for similarity evaluation between the pseudo-optical image and the real image.

(1) The GAN loss function is adopted to ensure that the generated pseudo-optical image is as close as possible to a real optical image. The pseudo-optical image and the corresponding real optical image are passed into the structure and texture branches, respectively, of the discriminator to ensure the consistency of the structure and texture. The GAN loss can be expressed as:

$$l_{GAN}=mi{n}_{G}ma{x}_D\mathbb{E}\left[\log D\left(Y_{real},Y_{real}^{Edge}\right)\right]+\mathbb{E}\left[\log 1-D\left(Y_{pseudo},Y_{pseudo}^{Edge}\right)\right]$$

(19)

Structure Loss: We define the structure loss by comparing the structure features generated by the structure generator with the structure features of the real optical image.

(1) The MSE loss function is adopted to ensure that the structure features generated by the structure generator are close to those of a real optical image. The texture MSE loss can be expressed as:

$$l_{MSE}^{Struture}=\mathbb{E}\left[\Vert f_s-Y_{pseudo}^{Edge}\Vert {}_2\right]$$

(20)

Texture Loss: Distinct from the reconstruction loss, we define the texture loss by comparing the texture features generated by the texture generator with the texture features of the real optical image.

(1) The MSE loss function is adopted to ensure that the texture features generated by the texture generator are close to those of a real optical image. The texture MSE loss can be expressed as:

$$l_{MSE}^{Texture}=\mathbb{E}[\Vert f_t-Y_{pseudo}\Vert {}_2]$$

(21)

(2) The FFL function is adopted to reduce the differences between the texture features generated by the texture generator and those of a real optical image in the frequency domain, and to reduce the artefacts in the texture features:

$$F{(u,v)}_{f_t}^Y={\sum }_{x=0}^{H-1}{\sum }_{y=0}^{W-1}f(x,y)·e^{-i2\pi \left(\frac{ux}{H}+\frac{vy}{W}\right)}$$

(22)

$$F{(u,v)}_{\mathrm{real}}^Y={\sum }_{x1=0}^{H-1}{\sum }_{y1=0}^{W-1}f\left(x_1,y_1\right)·e^{-i2\pi {\left(\frac{ux}{H}+\frac{vy}{W}\right)}_{}}$$

(23)

$$l_{FFL}^{Texture}=\frac{1}{HW}{\sum }_{u=0}^{H-1}{\sum }_{v=0}^{W-1}w(u,v){\left|F{(u,v)}_{f_t}^Y-F{(u,v)}_{\mathit{real}}^Y\right|}^2$$

(24)

where $F{\left(u,v\right)}_{f_t}^Y$ denotes the frequency value of the texture features generated by the texture generator, $F{\left(u,v\right)}_{real}^Y$ denotes the corresponding frequency value of the real optical image, (u, v) represents the coordinates of the spatial frequency in the frequency spectrum, H × W denotes the size of the image, (x, y) denotes the coordinates of an image pixel in the spatial domain, f(x, y) is the corresponding pixel value, and $w\left(u,v\right)$ denotes the dynamic spectral weight matrix.

In summary, the total loss is written as:

$$\begin{array}{cc}\hfill L& ={\lambda }_1\left(l_{MSE}^{rec})+{\lambda }_2(l_{FFL}^{rec}\right)+{\lambda }_3{l}_{VGG}^{rec}+{\lambda }_4{l}_{Style}^{rec}+{\lambda }_5{l}_{GAN}\hfill \\ \hfill & +{\lambda }_6\left(l_{MSE}^{Struture}+l_{MSE}^{Texture}\right)+{\lambda }_7\left(l_{FFL}^{Texture}\right)\hfill \end{array}$$

(25)

where ${\lambda }_1$, ${\lambda }_2$, ${\lambda }_3$, ${\lambda }_4$, ${\lambda }_5$, ${\lambda }_6$, and ${\lambda }_7$ are weighting coefficients of the loss functions, and the values set for our experiments are ${\lambda }_1=10$, ${\lambda }_2=50$, ${\lambda }_3=0.1$, ${\lambda }_4=250$, ${\lambda }_5=0.1$, ${\lambda }_6=1$, and ${\lambda }_7=5$.

3. Experiments

To demonstrate the effectiveness of our method, comparative experiments with Pix2pix [32], CycleGAN [33], S-CycleGAN [34], and EPCGAN [16] are presented. The results of qualitative visualizations show that our method achieves the best results in terms of both structure and texture. In quantitative experiments, three image quality assessment (IQA) metrics are used, namely, the peak signal-to-noise ratio (PSNR), the structural similarity (SSIM) [48], and the chromatic feature similarity (FSIMc) [49]. A higher PSNR indicates higher image quality, and the SSIM and FSIMc reflect the similarity between the pseudo-optical and real optical images, taking a value of 1 if the two images are identical. Experiments show that our method improves the PSNR by 21.0%, the FSIMc by 6.9%, and the SSIM by 161.7% in terms of the average metric values on all test images compared with the next best results, and the considerable SSIM improvement, in particular, proves the superiority of our dual-generator translation network in producing pseudo-optical images with better structure features.

We also present ablation experiments to demonstrate the effectiveness of the adopted loss functions. The superiority of the loss functions is demonstrated by qualitative visualization results that show the gradual texture and structure enhancement of the pseudo-optical images. In addition, quantitative experimental results show that adding the MSE loss function to the method presented in [38] can improve the PSNR by 2.3%, the FSIMc by 1.5%, and the SSIM by 13.9% in terms of the average metric values on all test images, whereas adding the FFL function can similarly improve the PSNR by 4.6%, the FSIMc by 0.7%, and the SSIM by 6.9%) on average, thus proving the superiority of these loss functions.

To verify the improvement in the matching performance, the translation results of our network are applied in keypoint detection and matching experiments, and experiments are presented to compare the performance before and after image translation. These experiments show that our method can improve the overall repeatability of image keypoint detection before and after translation by 111.7% on average for different keypoint detection methods and Euclidean distance thresholds, and the matching performance is also greatly improved. In the following subsections, we give the details of the comparative, ablation, and matching experiments.

3.1. Implementation Details

3.1.1. Datasets

The SEN1-2 datasets [50] contain 282384 optical and SAR images from all parts of the world and all meteorological seasons, and the size of each image block is 256 × 256 pixels. These data can be used for SAR-to-optical translation tasks. Similar to the research in [16], we selected a training set consisting of 2100 images from SEN1-2 that depict many kinds of land cover, such as mountains, forests, lakes, rivers, buildings, farmland, and roads. Figure 5 shows some of the image blocks selected as training samples. At the same time, we selected 222 images as the test set. The test images from SEN-1 and SEN-2 were only used to validate the performance of the proposed method, and were not used during training. The test SAR images were used for SAR-to-optical translation. The optical image blocks in our test set were only used to calculate the image quality of the results of the proposed method for SAR-to-optical translation. Figure 6 shows some of the image blocks selected as test samples. The same dataset was used for retraining in all comparative experiments to effectively evaluate the robustness of the network on the same dataset.

3.1.2. Training Details

Our experiments were based on the deep learning framework PyTorch [51]. The GPU used was an NVIDIA GTX3090, and the experimental operating system environment was Windows 10. During training, the Adam optimization algorithm [52] was used to train the network model. First, a learning rate of 2 × ${10}^{-4}$ was used in the initial training stage; then, once the model stabilized, the learning rate was adjusted to 5 × ${10}^{-5}$ to fine tune the model parameters, and the gradient of the batch normalization layer was frozen at the same time. The generator learning rate was 10 times that of the discriminator learning rate, which was 2 × ${10}^{-5}$. It took 10 h to train our model on the dataset using a batch size of 1. In the comparative experiments, the Adam optimizer with β1 = 0.5 and β2 = 0.999 was used for the optimization of the Pix2pix, CycleGAN, S-CycleGAN, and EPCGAN models. Specifically, the generators and discriminators were trained using the Adam optimizer for 200 epochs, and at 100 epochs, the learning rate began to be linearly reduced to 0.

3.2. A Comparison of Textural and Structural Information

To demonstrate the superiority of our proposed method, Figure 7 presents a visual comparison of the different SAR-to-optical translation methods. We can see that, as is evident from Test 2 in Figure 7, Pix2pix fails to preserve the details in the pseudo-optical image, with some shift in style and poor results in terms of both texture and structure. The CycleGAN results are visually superior to those of Pix2pix, containing better texture and structure information, but the details of the structure are blurred and unclear. S-CycleGAN produces images that are relatively clear in texture, but the structure features are distorted. EPCGAN adds gradient information branching on the basis of S-CycleGAN; as a result, its structure information is improved, but the gradient information and the backbone network cannot be fully coupled, and structure or texture features may be sacrificed to some extent to fit the model, which leads to poor results. In contrast, our proposed method yields the best texture and structure information in Tests 1–4.

3.3. Results and Analysis

We quantitatively evaluated Pix2pix [32], CycleGAN [33], S-CycleGAN [34], EPCGAN [16], and our method using the Test 1–4 data to demonstrate the superiority of our method. In addition, we computed the mean values of the PSNR, SSIM, and FSIMc metrics for 222 pairs of test images to further demonstrate the strong robustness of our method. The experimental results show that our method achieves large improvements in Tests 1–4, improving the PSNR by 21.0%, the FSIMc by 6.9%, and the SSIM by 161.7% in terms of the average metric values on all test images compared with the next best results, as shown in

Table 2. IQA results for different methods obtained by averaging the evaluation metrics over 222 pairs of images in the test set. The best values for each evaluation index are shown in bold.

IQA	DATA	Pix2pix	CycleGAN	S-CycleGAN	EPCGAN	Ours
PSNR	Test 1	11.2090	11.8234	13.6493	13.1767	19.0867
	Test 2	12.0022	13.0566	14.3403	15.7064	20.2105
	Test 3	13.1578	12.8568	16.5878	16.6274	20.1606
	Test 4	11.6502	14.7996	14.1882	16.0283	20.2383
	all_test (Average)	11.2212	12.0270	14.4801	14.7253	17.8228
FSIMc	Test 1	0.5962	0.6062	0.6262	0.6210	0.7357
	Test 2	0.5980	0.6071	0.6712	0.6623	0.7736
	Test 3	0.5222	0.5622	0.6651	0.6859	0.7793
	Test 4	0.5383	0.7011	0.6942	0.7005	0.7837
	all_test (Average)	0.5719	0.6055	0.6699	0.6611	0.7167
SSIM	Test 1	0.0711	0.0413	0.0566	0.0746	0.4574
	Test 2	0.0825	0.0642	0.0909	0.1318	0.4586
	Test 3	0.0567	0.0601	0.2367	0.2326	0.4911
	Test 4	0.0616	0.2023	0.1697	0.2042	0.4263
	all_test (Average)	0.0528	0.0533	0.1204	0.1264	0.3308

.

The main reasons for these findings are as follows. The performance of Pix2pix is relatively weak because it has less constrained loss conditions, whereas CycleGAN, which is based on the idea of cycles with more strongly constrained SAR-to-optical translation, produces better texture and structure features by virtue of the addition of cyclic loss functions. However, CycleGAN still faces problems with image translation and artefacts. S-CycleGAN improves the CycleGAN network by adding an MSE loss function to solve the above problems. However, the NRDs-induced differences between SAR and optical images still lead to poor structural information in the generated pseudo-optical images. To address this shortcoming, EPCGAN incorporates gradient information to guide the SAR-to-optical translation process, thereby improving the structural similarity and visual effect. Compared with the other methods, our proposed dual-generator translation network fuses texture and structure information to achieve considerable enhancement of both the texture and structure features, thus achieving the best performance in these tests.

3.4. Ablation Experiment

In our proposed method, we incorporated multiple loss functions to improve the translation quality. To demonstrate the superiority of our method, we conducted ablation experiments to demonstrate the effects of the different loss functions, as manifested in the variations of the three evaluation metrics. The gradual improvement in the texture and structure of the pseudo-optical images, as shown in the qualitative visualization results in Figure 8, proves the superiority of adding these loss functions. The quantitative experimental results further show that adding the MSE loss function can improve the PSNR by 2.3%, the FSIMc by 1.5%, and the SSIM by 13.9% in terms of the average metric values on all test images, whereas adding the FFL function can further improve the PSNR by 4.6%, the FSIMc by 0.7%, and the SSIM by 6.9% on average, as shown in

Table 3. IQA results in the ablation experiment obtained by averaging the evaluation metrics over 222 pairs of images in the test set. The best values for each evaluation index are shown in bold.

IQA	DATA	Guo [38]	Ours (+MSE Loss)	Ours (+MSE Loss +FFL Loss)
PSNR	Test 5	22.6563	22.8161	23.3868
	Test 6	13.0547	13.8798	16.4298
	Test 7	16.4138	16.5352	19.4805
	Test 8	22.5349	22.7283	24.8288
	all_test (Average)	16.6327	17.0269	17.8228
FSIMc	Test 5	0.7227	0.7255	0.7271
	Test 6	0.6789	0.7026	0.7231
	Test 7	0.7745	0.7837	0.8068
	Test 8	0.7747	0.7734	0.7824
	all_test (Average)	0.7007	0.7117	0.7167
SSIM	Test 5	0.3237	0.3344	0.3756
	Test 6	0.2493	0.3038	0.3818
	Test 7	0.3534	0.4079	0.4459
	Test 8	0.4897	0.5259	0.5364
	all_test (Average)	0.2714	0.3092	0.3308

.

3.5. Matching Applications

To further verify the superiority of our proposed algorithm, the results of comparative experiments on the repeatability of keypoint detection and matching between real optical and generated pseudo-optical (O-PO) images and between real optical and real SAR (O-S) images with different Euclidean distance thresholds and different keypoint detection methods are shown in

Table 4. O-PO: real optical images and generated pseudo-optical images; O-S: real optical images and real SAR images. Experimental comparison of O-PO and O-S image keypoint repeatability under different Euclidean distance thresholds and keypoint detection methods.

Keypoint Repeatability	Keypoint Detection Methods	Optical Keypoint Number	PO/S Keypoint Number	Translation Mode	Euclidean Distance Threshold (L2)
					3.0	2.5	2.0	1.5	1.0
Test 1 % Rep.	SuperPoint [54]	603	488/288	O-PO	56.64%	45.55%	30.25%	20.18%	9.93%
				O-S	26.23%	19.25%	11.07%	7.46%	4.09%
	Key.Net [55]	494	593/419	O-PO	40.66%	36.40%	28.07%	22.37%	15.29%
				O-S	19.27%	14.68%	9.20%	5.91%	3.72%
	SIFT [6]	564	600/597	O-PO	51.20%	42.27%	34.36%	23.19%	14.95%
				O-S	39.28%	29.80%	22.22%	13.61%	5.50%
	SURF [7]	530	517/542	O-PO	53.30%	46.03%	36.87%	26.93%	17.38%
				O-S	35.45%	24.63%	16.79%	10.26%	5.41%
	BRISK [56]	613	604/585	O-PO	67.05%	61.13%	49.96%	39.77%	24.65%
				O-S	39.23%	30.55%	21.70%	14.02%	6.51%
	Harris [57]	587	599/576	O-PO	50.08%	38.11%	27.65%	18.21%	9.10%
				O-S	40.24%	30.26%	18.57%	10.83%	4.64%
	PC-Harris [58]	532	458/581	O-PO	53.13%	43.23%	30.30%	19.80%	9.90%
				O-S	40.61%	29.11%	18.86%	9.88%	5.03%
	Hessian	569	563/588	O-PO	55.83%	44.52%	33.92%	22.61%	11.84%
				O-S	41.83%	33.36%	24.20%	12.96%	5.36%

. The higher the keypoint repeatability is, the higher the likelihood that the extracted keypoints are correctly matched [53]. From the results, it can be seen that our method can be applied for keypoint detection well. The smaller the Euclidean distance threshold is, the more obvious the improvement in keypoint repeatability. The average improvements for different keypoint detection methods are 58.4%, 77.6%, 104.1%, 138.6%, and 179.9% for Euclidean distance thresholds of 3.0, 2.5, 2.0, 1.5, and 1.0, respectively. Overall, the average repeatability of image keypoint detection before and after translation is improved by 111.7%, indicating that our translation method can substantially improve the number of potential matching points and further improve the root mean square error (RMSE) of correctly matched point pairs.

We use the number of correct matches (NCM) and the RMSE to quantify the improvement in matching performance. The experimental results for the O-PO and O-S image matching performance on the images from Tests 1–4 under different matching methods are compared in

Table 5. O-PO: real optical images and generated pseudo-optical images; O-S: real optical images and real SAR images. Experimental comparison of O-PO and O-S matching performance under different matching methods.

Match Pairs	Translation Mode	LoFTR [9]		RIFT [11]		PSO-SIFT [59]		SAR-SIFT [60]		SIFT [6]
		NCM	RMSE	NCM	RMSE	NCM	RMSE	NCM	RMSE	NCM	RMSE
Test 1	O-PO	638	0.1113	232	1.6275	75	1.5361	22	0.5712	29	0.6069
	O-S	0	/	86	1.8802	0	/	0	/	0	/
Test 2	O-PO	485	0.1861	328	1.7807	69	1.5554	18	0.5427	26	0.5795
	O-S	0	/	131	1.8214	0	/	0	/	0	/
Test 3	O-PO	335	0.6239	211	1.8689	20	1.3701	7	0.4694	14	0.5907
	O-S	0	/	91	1.9132	0	/	0	/	0	/
Test 4	O-PO	404	0.2441	265	1.8538	57	1.7190	8	0.6512	15	0.5801
	O-S	72	1.1113	180	1.9114	11	1.0988	0	/	8	0.4473

. It can be seen that the pseudo-optical images obtained using our method for heterologous source image matching have high application value. In matching experiments using the radiation-variation insensitive feature transform (RIFT), as shown in Figure 9, there are improvements in both the NCM and RMSE. Specifically, the average NCM increases by 137 on the Test 1–4 data, and the average RMSE accuracy improves by 5.25%. The experimental results obtained using the scale-invariant feature transform (SIFT), position scale orientation SIFT (PSO-SIFT), LoFTR [9], and SAR-SIFT also show that our method successfully converts heterologous images that cannot be correctly matched into pseudo homologous images that can be correctly matched, providing a new approach for solving the problem of matching heterologous remote sensing images.

4. Discussion

There are large NRDs between optical and SAR images in terms of both structure and texture, which pose a great challenge for optical–SAR image translation. In most image translation tasks, there is a strong connection between the source and target image domains; for example, the structural information may be identical, with the only differences appearing in texture and colour. In contrast, in optical–SAR image translation, both texture and structure information need to be considered. The existing methods all have the problem of favouring either texture features or structure features, and the two cannot influence each other, which can lead to serious structure or texture distortion; furthermore, simply increasing the gradient information of SAR images cannot yield good results. To overcome these challenges, our method considers the dual generation of texture and structure, and fuses the deep structure and texture features thus obtained to produce pseudo-optical images with clear structure and texture information (see Figure 7).

To qualitatively evaluate the structural features of the pseudo-optical images generated by our network, inspired by the research in [61], a fast Fourier transform (FFT)-accelerated sum of squared differences (SSD) method is used to measure the similarity between the structural features of pseudo-optical images obtained via different methods and those of real optical images. The value of the SSD score plot indicates the offset between image pairs, and a smaller value indicates a higher similarity of their features [62]. In addition, it is noted that the SSD score map obtained with the maximum offset set to 8 pixels has dimensions of 17 × 17. For clarity of observation, we set the maximum offset value to 8 pixels. As shown in Figure 10, the structural features of the pseudo-optical image obtained using our proposed method have the highest similarity with the structural features of the real optical image. Because our network has two generators, one for texture and one for structure, the texture and structure features are coupled and provide feedback to each other to enhance the edge information. Moreover, we add an MSE loss function and an FFL function to reduce the difference between pseudo-optical and real optical images in both the spatial and frequency domains to achieve greater structural enhancement. Consequently, our method can produce pseudo-optical images with significantly improved structural features.

The use of our translation results in matching applications can substantially improve the matching performance for SAR and optical images, enhancing the number of potential matching points obtained through keypoint detection, and significantly improving the number of correctly matched point pairs. The practical value of high-precision SAR-to-optical translation for SAR–optical image matching has been explored, and the necessity of high-precision SAR-to-optical translation networks for image matching has been demonstrated. However, in the practical application of SAR images, motion error is a key problem that needs to be solved, and the presence of motion error will lead to unfocused SAR images [63]. Edge features and texture features cannot be obtained, which leads to the poor robustness of our method. In the future, we will work on improving the robustness of the algorithm to apply it in practical applications.

5. Conclusions

In this paper, considering the NRDs in the texture and structure features of optical and SAR images, we summarize the current methods of SAR-to-optical image translation and propose a dual-generator translation network fusing structural and texture features for SAR–optical image matching. Comparative experiments with the latest methods and ablation experiments are conducted to demonstrate that our method achieves superior performance in SAR-to-optical translation. Our method improves the PSNR by 21.0%, the FSIMc by 6.9%, and the SSIM by 161.7% in terms of the average metric values on all test images compared with the next best results. Furthermore, the ablation experiments demonstrate that our introduction of an MSE loss function and an FFL function can effectively reduce the spatial- and frequency-domain differences between pseudo-optical images and real optical images and enhance the visual quality of the generated pseudo-optical images, especially in regard to texture, structure, and color information. In addition, to further demonstrate the superiority of our method, comparative experiments of keypoint detection and matching in heterologous remote sensing images before and after translation are presented, and the results prove that the proposed high-precision image translation method can significantly improve the matching performance for heterologous remote sensing images. Our method improves the average keypoint repeatability by approximately 111.7% and the matching accuracy by approximately 5.25%. In the future, we will strive to further improve the accuracy of our model and enhance its generalization ability.