Infrared Image Generation Based on Visual State Space and Contrastive Learning

Abstract

The preparation of infrared reference images is of great significance for improving the accuracy and precision of infrared imaging guidance. However, collecting infrared data on-site is difficult and time-consuming. Fortunately, the infrared images can be obtained from the corresponding visible-light images to enrich the infrared data. To this end, this present work proposes an image translation algorithm that converts visible-light images to infrared images. This algorithm, named V2IGAN, is founded on the visual state space attention module and multi-scale feature contrastive learning loss. Firstly, we introduce a visual state space attention module designed to sharpen the generative network’s focus on critical regions within visible-light images. This enhancement not only improves feature extraction but also bolsters the generator’s capacity to accurately model features, ultimately enhancing the quality of generated images. Furthermore, the method incorporates a multi-scale feature contrastive learning loss function, which serves to bolster the robustness of the model and refine the detail of the generated images. Experimental results show that the V2IGAN method outperforms existing typical infrared image generation techniques in both subjective visual assessments and objective metric evaluations. This suggests that the V2IGAN method is adept at enhancing the feature representation in images, refining the details of the generated infrared images, and yielding reliable, high-quality results.

1. Introduction

Due to the limitations of application backgrounds and support capabilities, the reference images in infrared imaging guidance are usually visible-light images, while the real-time images are infrared images. The imaging mechanisms of infrared and visible-light images are different, leading to significant feature differences between them, which, in turn, increases the difficulty of scene matching in infrared imaging guidance [1]. Using infrared image simulation technology to generate the infrared characteristics of scenes in the required environment can not only effectively reduce the cost of obtaining infrared data, but also help to generate infrared data that are difficult to obtain in field tests under many natural environmental and scene conditions [2]. In addition to enriching infrared data, the method of converting visible-light images to infrared images can also be used to address the issues related to heterogeneous image matching and the integration of heterogeneous images. The generated infrared data can be applied to various fields such as aviation [3], navigation [4,5,6,7,8], meteorology [9,10,11,12], geology [13,14,15,16], and agriculture [17,18,19,20], providing basic and reliable data for tasks such as detection [21,22,23,24,25], classification [26,27,28,29], positioning [30,31,32,33], recognition [34,35,36,37], and tracking [38,39,40].

The current research on infrared mainly focuses on the mid-wavelength and long-wavelength ranges because the energy radiated from the ground is primarily concentrated in the mid-infrared and far-infrared spectral bands. Currently, the mainstream infrared image simulation technologies can be divided into two major categories: one is based on infrared characteristic modeling image simulation technology [41], and the other is based on deep learning infrared image simulation technology [2]. The former is based on the establishment of mathematical models related to the infrared characteristics of the scene and displays the infrared simulation effect of the scene through computer simulation. The latter uses deep learning algorithms to train a large number of visible-light scenes and their infrared characteristic effects, obtaining a mapping model between the visible-light and infrared images of the scene. Then, the infrared characteristics of the visible-light scene can be generated according to this model and displayed in the form of an image.

Infrared simulation technology based on infrared characteristic modeling can generate high-quality target infrared textures. However, these methods have limitations in generating infrared characteristics of target scenes in the presence of disturbances such as smoke and shadows, and in batch processing of data. They also exhibit lower levels of automation. Deep learning plays a significant role in feature extraction and mapping fitting, providing effective means to improve the quality of the generated images. Deep generative models are an important component in deep learning-based image generation, with Generative Adversarial Networks (GANs) [42] demonstrating superior performance in image transformation compared to other generative models.

Based on generative adversarial networks, the translation from visible-light to infrared images can be categorized into paired and unpaired types according to the data type. The visible-light image to infrared image conversion methods based on the Pix2Pix [43] framework are used for paired data, such as ThermalGAN [44], LayerGAN [45], InfraGAN [46], IR-GAN [47], and TMGAN [48]. ThermalGAN [44] and LayerGAN [45] generate infrared images through multimodal data generation. ThermalGAN generates infrared images in two steps: first, it generates the target’s average-temperature infrared image using visible-light images and temperature vectors, and then it generates a more refined infrared image using the target’s average-temperature infrared image and visible-light images. LayerGAN includes two methods for generating infrared images. One method uses temperature vectors, semantic segmentation images, and thermal segmentation images to generate infrared images. The other method uses visible-light images, semantic segmentation images, and thermal segmentation images to generate infrared images. Although these two works are of great significance for pedestrian re-identification tasks based on infrared images, they have high data requirements and require multimodal data. InfraGAN [46] and IR-GAN [47] address the issue of edge distortion in the generated infrared images by constructing new generative networks and loss functions to strengthen the constraints on the edge information of the generated infrared images. However, they do not delve deeply into the feature mapping relationship between visible-light images and infrared images.

Based on the CycleGAN [49] framework, visible-light-to-infrared image conversion models are used for unpaired data. Examples include SIR-GAN [50], DAGAN [51], and CVIIT [52]. SIR-GAN [50] takes traditional infrared simulation data as input to obtain infrared images with rich texture information. DAGAN [51] uses a novel dual-attention mechanism for the automatic prediction of flashovers in compartment fires using visible-light-image-to-infrared translation. CVIIT [52] introduced a contrastive loss to ensure the consistency of content between the generated image and the source image, which can enhance the quality of image conversion under conditions of diminished illumination. The Edge-guided Multi-domain RGB-to-TIR image Translation algorithm (EMRT) [53] preserves more detail in the infrared image by constraining the consistency of edge information between the generated image and the source image. Although these methods have relaxed the requirements for data, they generally produce inferior infrared image results compared to paired methods. Therefore, the focus of this paper is on the translation of visible-light images to infrared images for paired data. Paired visible-light and infrared images capture the same ground objects, yet their different imaging mechanisms result in distinct data characteristics for these objects. Consequently, establishing an accurate feature mapping relationship is crucial for improving the quality of the infrared-generated images. A typical process of visible-light-image-to-infrared-image translation is shown in Figure 1.

Although Generative Adversarial Networks (GANs) have shown success in converting visible-light images to infrared images, they still have some shortcomings. The main issues include the distinct imaging mechanisms of visible-light and infrared images, which result in reduced accuracy in feature mapping by GANs, leading to suboptimal fitting effects. Furthermore, the generated infrared images suffer from issues such as poor texture consistency and detail loss.

To address the aforementioned challenges, this work proposes a method for generating infrared images from visible light, named V2IGAN, which is based on visual state space and multi-scale feature contrast learning. Utilizing the conditional generative adversarial network (CGAN) [54] framework, V2IGAN incorporates a visual state space attention module into its generative network. The module establishes feature mappings between visible-light and infrared images using a state-space model, emphasizing key regions in the visible-light images. Furthermore, V2IGAN employs a multi-scale feature contrast learning loss to ensure consistency across scales in the generated infrared images, thereby reducing the loss of detail. Notably, this strategy leverages the discriminator’s attention maps to enhance learning for challenging samples. The visual state space attention module and multi-scale feature contrast learning loss synergize to enhance the production of high-quality infrared images.

The main contributions are as follows:

• A visual state space attention module is introduced into V2IGAN to focus the generative network on key areas within visible-light images effectively.
• A multi-scale feature contrast learning function is considered in the present method to ensure semantic consistency between the generated and target images effectively.
• Experiments conducted on large benchmark datasets across various scenarios demonstrate that V2IGAN is capable of producing high-quality, reliable infrared images.

The rest of this article is organized as follows. Section 2 briefly reviews the related work. In Section 3, the methodology of the proposed method, V2IGAN, is described in detail. The experiments and results are presented in Section 4, and Section 5 presents the discussion and analysis. Concluding remarks are given in Section 6.

2. Related Work

2.1. Generative Adversarial Networks

The basic idea of Generative Adversarial Networks (GANs) [42] is to set up a zero-sum game between two players, i.e., the generator and the discriminator. The generator creates samples that are intended to come from the same distribution as the training data. The discriminator examines samples to determine whether they are real or fake. The discriminator learns using supervised learning techniques, dividing inputs into two classes (real or fake). The generator is trained to fool the discriminator.

To overcome the limitation of original GANs in generating images with specific attributes, Mirza et al. [54] proposed the concept of Conditional Generative Adversarial Networks (CGANs). CGANs provide precise control over image generation by incorporating conditional variables. In recent years, methods based on the CGAN architecture have achieved significant advancements in enhancing image quality and diversity [8,55], improving user interactivity, and boosting model performance. The latest research encompasses not only sample-based transformation methods, language-driven models, and techniques for generating realistic images from abstract sketches but also includes in-depth explorations of the principles of CGAN algorithms [56,57], systematic developments in GAN architecture, and studies on stability and scalability in large-scale training. This work introduces visible-light images as conditional information to constrain and control the generation of corresponding infrared images.

2.2. Visible-to-Infrared Image Translation Based on CGAN

The process of Image-to-Image (I2I) translation [43], based on Conditional Generative Adversarial Networks (CGANs), can facilitate the transformation of images from the source domain (i.e., visible-light image) to the target domain (i.e., infrared image) by training a sophisticated deep learning model. Common models such as ThermalGAN [44], Pix2Pix-MRFFF [58], InfraGAN [46], and IR-GAN [47] are adept at converting visible-light images into their infrared counterparts. The superior performance of these models is largely based on the following three keys.

Network Architecture Enhancements: The integration of ResNet [59], UNet [60], and attention mechanisms into the network architecture significantly boosts the performance of the generative networks. For instance, EPSANet [36] incorporates an efficient pyramid squeeze attention module aimed at extracting multi-scale spatial information at a more granular level and developing a long-range channel dependency.

Adversarial Network Utilization: The deployment of adversarial networks with the refined image perception capabilities, such as the PatchGAN [61], MultiPatchGAN [62], and UNetGAN [63] models, further enhances the quality of the generated infrared images.

Stabilization Techniques: For the stabilization of the training process, models such as the Least Squares GAN (LSGAN) [64] and Wasserstein GAN (WGAN) [65] have been proposed to provide more stable and less mode-collapse-prone training dynamics.

In terms of loss functions, these models typically integrate traditional L1 and GAN losses with advanced loss functions specifically designed to enhance the edge information in infrared images. This includes the application of the structural similarity index measure (SSIM) loss [46] and gradient vector loss [47], both of which are aimed at improving the sharpness and definition of edges in the generated images. In conclusion, the translation from visible-light images to infrared images is a complex and multifaceted challenge in the field of computer vision, necessitating an in-depth analysis of the relationship between the two types of images.

2.3. Contrastive Learning

Contrastive learning has demonstrated strong advantages in state-of-the-art unsupervised visual representation learning tasks by learning a mapping function that continuously “pulls closer” the distances with related samples and “pushes away” the distances from other samples. These related samples are called positive samples, and other samples are called negative samples [66,67]. In contrastive learning methods, the approach to constructing positive and negative samples correctly is crucial. Recently, some studies have explored the application of contrastive learning in image translation [68,69].

TUNIT [70] uses a contrastive loss to simultaneously separate image domains and transform the input image to the target domain. DivCo, on the other hand, uses a contrastive loss to properly constrain the positive and negative relationships between specified generated images in the latent space. CUT [71] employs a noise-contrastive estimation framework to maximize the mutual information between input and output, thereby improving the performance of unsupervised image transformation models. To further leverage contrastive learning, DCLGAN [72] extends the unidirectional mapping to bidirectional mapping, performing better in learning embeddings. EnCo [73] requires only a set of encoders and decoders, and the target images do not need to be re-encoded to overcome the issue of blurred infrared images generated by existing algorithms. This study employs contrastive learning in infrared image generation to enhance the fine-grained details of the generated infrared images.

3. Proposed Method

The V2IGAN algorithm architecture proposed in this work mainly consists of a generator network, translating visible-light images into infrared images, and a discriminator network designed to distinguish between real infrared images and generated infrared images. Figure 2 illustrates the framework flowchart of the V2IGAN algorithm. The generator network features an encoder–decoder architecture, while the discriminator network is structured around the PatchGAN model. Furthermore, the V2IGAN algorithm incorporates a visual state space attention module and employs a multi-scale feature contrast learning loss.

3.1. Proposed Network Architecture

Generator: Infrared and visible-light images have distinct imaging mechanisms, leading to significant differences in texture and edge features. These differences elevate the challenge of translating visible-light images into infrared images. The Visual State Space (VSS) block capitalizes on the strengths of State Space Models (SSMs) in visual data processing. It features a global perceptual field, input-dependent weighting parameters, and a linear computational complexity, which effectively extracts features from visible-light images. Compared to the attention mechanism based on Transformer, the VSS module has a higher computational efficiency and requires less hardware infrastructure. The performance of the VSS module on multiple visual tasks, such as image classification, object detection, and semantic segmentation, has demonstrated its superiority. Building upon the performance advantages of the VSS block, this work presents a VSS attention module, depicted in Figure 3b. This module refines the generative network’s focus on key regions within the visible-light image, thereby fortifying the feature extraction effectiveness. Consequently, it improves the model’s capacity to accurately model feature relationships.

The generator of the V2IGAN algorithm, as illustrated in Figure 3a, comprises three parts: a down-sampling module, a VSS attention module, and an up-sampling module. The operation of the generator can be described as follows. The input visible-light image x initially undergoes two successive down-sampling operations, diminishing the image size while simultaneously augmenting the number of channels, thereby capturing the essential features of the visible-light image. The VSS attention module then refines these features to accentuate the features of interest within the visible-light image. Finally, through two sequential up-sampling stages, the image size is progressively enlarged, and the number of channels is correspondingly reduced, ultimately resulting in the reconstruction of the image into an infrared image $\widehat{y}$.

Discriminator: V2IGAN employs a discriminator constructed by PatchGAN. The Markovian discriminator is a type of discriminative model composed of convolutional layers, with the output being a matrix of size $N\times N$, and the average value of this matrix is taken as the output for true/false. Since each value in the output matrix corresponds to a receptive field in the original image, that is, a receptive field corresponds to a regional block (patch) in the original image, GANs with this structure are also known as PatchGAN [61].

Taking an image with a size of $256\times 256$ as an example, conventional GANs map the $256\times 256$ image to a single scalar output, representing “true” or “false”. However, this does not easily reflect the local features of the image. In contrast, PatchGAN maps the $256\times 256$ image to a matrix of size $N\times N$, where each $N\times N$ represents whether the regional block in the image is true or false. PatchGAN achieves the extraction and representation of local image features, allowing for the model to pay more attention to the detailed information of the image, which is conducive to generating higher-quality images.

3.2. VSS Attention Module

The visible-light image is first down-sampled to obtain $f_x$. The input of the VSS attention module is then processed through adaptive average pooling and adaptive max pooling to obtain $Avg\left(f_x\right)$ and $Max\left(f_x\right)$, respectively. After the convolution process, these are respectively multiplied by the original input to obtain $f_x\ast Conv\left(Avg\left(f_x\right)\right)$ and $f_x\ast Conv\left(Max\left(f_x\right)\right)$, respectively. The results are then multiplied by the concatenated features $Conv\left(Cat\left(Avg\left(f_x\right)andMax\left(f_x\right)\right)\right)$ after convolution and added to the input VSS block. The final output of the VSS attention module is $VSS(Conv\left(Cat(Avg\left(f_x\right),Max\left(f_x\right))\right)\ast f_x\ast Conv\left(Avg\left(f_x\right)\right)+Conv\left(Cat(Avg\left(f_x\right),Max\left(f_x\right))\right)\ast f_x\ast Conv$ $\left(Max\left(f_x\right)\right))$.

The VSS attention module includes a crucial component called the VSS block. The VSS block, derived from the VMamba [74], represents a pivotal component of the generative network, as illustrated in Figure 4. After undergoing the layer normalization process, input data are bifurcated into two pathways. In the side pathway, input data are processed by a linear layer followed by a block with the SiLU activation function. In the main pathway, input data are first processed by a linear layer, a depthwise separable convolution block, and a SiLU-activation-function block, and then directed by the 2D-Selective-Scan (SS2D) module for enhanced feature extraction. Next, layer normalization is performed on the extracted features, which are then merged with the side pathway’s results through pixel-wise multiplication. Finally, the features are mixed using a linear layer, and the output features are combined with the input through a residual connection to form the output of the VSS block.

The 2D selective scan (SS2D) consists of three parts, i.e., the scanning expansion operation, selective scan space state sequential model (S6), and the scanning fusion operation, as illustrated in Figure 5. As depicted in the figure, the scanning expansion operation unfolds an input image from the top-left to the bottom-right, from the bottom-right to the top-left, from the top-right to the bottom-left, and from the bottom-left to the top-right into sequences along four different directions. These sequences are processed by the S6 block for feature extraction, ensuring a thorough scan of information from all four directions, capturing a variety of features. Subsequently, as shown in Figure 5, the scanning fusion operation sums and merges the sequences from the four directions, restoring the output image to the same size as that of the input. The specific operation can be described as follows. Given the input feature s, the output feature $s_o$ of SS2D can be represented as follows:

$$s_d=expand(s,d),$$

(1)

$$\overline{s_d}=S6\left(s_d\right),$$

(2)

$$s_o=merge(\overline{s_{d1}},\overline{s_{d2}},\overline{s_{d3}},\overline{s_{d4}}),$$

(3)

where $d\in D=1,2,3,4$ are four different scanning directions. $expand(·)$ and $merge(·)$ correspond to the scan expansion as depicted in Figure 5 and scan merging as depicted in Figure 5. Refer to ref [74] for further details about S6. The pseudo-code for S6 is presented in Algorithm 1.

Algorithm 1 Pseudo-code for S6 in SS2D.

Input:  $s_d,\mathrm{the}\mathrm{input}\mathrm{feature}$ Params:  $\mathbf{A},\mathrm{the}\mathrm{nn}.\mathrm{Parameter}$                 $\mathbf{D},\mathrm{the}\mathrm{nn}.\mathrm{Parameter}$ Operator:  $Liner(·),\mathrm{the}\mathrm{linear}\mathrm{projection}\mathrm{layer}$ Output:  $\overline{s_d},\mathrm{the}\mathrm{output}\mathrm{featurer}$ $$ $\Delta ,\mathbf{B},\mathbf{C}=Liner\left(s_d\right),Liner\left(s_d\right),Liner\left(s_d\right)$ $\overline{\mathbf{A}}=exp\left(\Delta \mathbf{A}\right)$ $\overline{\mathbf{B}}={\left(\Delta \mathbf{A}\right)}^{-1}(exp\left(\Delta \mathbf{A}\right)-\mathbf{I})·\Delta \mathbf{B}$ $h_t=\overline{\mathbf{A}}{h}_{t-1}+\overline{\mathbf{B}}{s}_{dt}$ $\overline{s_{dt}}=\mathbf{C}{h}_{t1}+\mathbf{D}{s}_{dt}$ $\overline{s_d}=[\overline{s_{d1}},\overline{s_{d2}},\dots ,\overline{s_{dt}},\dots ,\overline{s_{dL}}]$ return   $\overline{s_d}$

3.3. Multi-Scale Feature Contrastive Learning

To enhance the fine-grained features of the generated images, this study introduces a contrastive learning-based strategy, as shown in Figure 6. Compared with the CUT [71] and DCLGAN [72], the proposed approach requires only one set of encoders and decoders and does not necessitate re-encoding of the images. Consequently, it demands fewer training resources.

The proposed algorithm features a generative network composed of an encoder and a decoder, with each having L stages. The input x of the encoder is represented by $h_0$. After passing through the encoder, a series of encoded features of varying sizes are obtained, represented by ${\left\{h_l\right\}}_1^L={\left\{G_{enc}^l\left(h_{l-1}\right)\right\}}_1^l$. These encoded features are then input into the decoder, which generates a corresponding series of decoded features, expressed as ${\left\{h_l\right\}}_{L+1}^{2L}={\left\{G_{dec}^l\left(h_{l-1}\right)\right\}}_{L+1}^{2L}$. It is important to note that the outputs from the corresponding layers in both the encoder and decoder are of equivalent dimensions. Specifically, $h_l$ and $h_{2L-1}$ have the same size, and they can be represented by $(h_l,h_{\overline{l}})$, which signifies a pair of encoding and decoding features with identical dimensions.

As shown in Figure 6, for any pair of encoding–decoding features $(h_l,h_{\overline{l}})$, a query vector is sampled from the output feature map’s position of the decoder’s $\overline{l}$ stage, and positive samples are sampled from the corresponding positions of the encoder and from N negative samples from other locations of the encoder. Then, using a two-layer multilayer perceptron (MLP) with a shared linear layer, the query vector, positive samples, and negative samples are transformed into a K-dimensional embedding space, and vectors $q,k^{+}\in {\mathbb{R}}^K$ and $k^{-}\in {\mathbb{R}}^K$ are obtained, respectively. To avoid the collapse of the contrastive learning mode, vectors $q,k^{+}\in {\mathbb{R}}^K$ and $k^{-}\in {\mathbb{R}}^K$ are mapped into a unit hypersphere using the L2 normalization. Contrastive learning focuses on putting the query vector closer to the positive samples but further away from the negative samples in the embedding space, which represents a classification problem of $(N+1)$ classes. Therefore, a cross-entropy function is used for calculation in this study. This function indicates the probability of selecting a negative sample as a positive sample, and it is defined as follows:

$$l_{CL}(q,k^{+},k^{-})=-log{\displaystyle \frac{exp(q·k^{+}/\tau )}{exp(q·k^{+}/\tau )+{\sum }_{k=1}^{N}exp(q·k^{-}/\tau )}},$$

(4)

where · denotes the dot product of vectors; $\tau$ is a temperature coefficient used to scale the distance between the query vector and the positive and negative samples. The temperature coefficient determines the degree to which the contrastive loss focuses on difficult negative samples. A higher temperature coefficient means that the model will not pay too much attention to more challenging negative samples; conversely, a lower temperature coefficient means that the model will pay more attention to difficult negative samples that have a high similarity with the sample, giving them a larger gradient to separate them from positive samples. Based on the CUT [71], DCLGAN [72], and CVIIT [52] algorithms, we set $\tau$ to 0.07.

Expanding its application to multi-scale features $(h_l,h_{\overline{l}})$ can be expressed as follows:

$${\mathsf{\ell }}_{CL}(G,H,X)=E_{x\sim X}\sum _l^L\sum _s^{S_l}l\left(q_{l,s},k_{\overline{l},s}^{+},k_{\overline{l},s}^{-}\right).$$

(5)

Contrastive learning-based methods typically employ a random negative sampling scheme, which, to some extent, neglects the focus on difficult samples. The discriminator’s attention map for generated images represents the quality of the generated image regions, thereby providing the location of difficult samples. Based on the discriminator’s attention map, the generative network’s focus on key areas is enhanced.

3.4. Loss Function

V2IGAN is divided into two versions suitable for paired data (V2IGAN-P) and unpaired data (V2IGAN-U). The V2IGAN-P algorithm employs L1 loss, contrastive learning loss, and LSGAN loss. The V2IGAN-U algorithm incorporates identity loss, contrastive learning loss, and LSGAN loss, with the identity loss being used to stabilize the network training. Their definitions are as follows:

$${\mathsf{\ell }}_{L1}=\left|\right|y-{G\left(x\right)\left|\right|}_1,$$

(6)

$${\mathsf{\ell }}_{id}=\left|\right|y-{G\left(y\right)\left|\right|}_1,$$

(7)

$${\mathsf{\ell }}_{CGAN}(G,D)=E_{y\in \mathcal{Y}}\left[{\left(D\left(y\right)\right)}^2\right]+E_{x\in \mathcal{X}}\left[{(1-D\left(G\left(x\right)\right))}^2\right],$$

(8)

$${\mathcal{L}}_{V2IGAN-P}={\lambda }_1{\mathsf{\ell }}_{CGAN}(G,D)+{\lambda }_2{\mathsf{\ell }}_{L1}+{\lambda }_3{\mathsf{\ell }}_{CL},$$

(9)

$${\mathcal{L}}_{V2IGAN-U}={\psi }_1{\mathsf{\ell }}_{CGAN}(G,D)+{\psi }_2{\mathsf{\ell }}_{id}+{\psi }_3{\mathsf{\ell }}_{CL},$$

(10)

where ${\mathcal{L}}_{V2IGAN-P}$ and ${\mathcal{L}}_{V2IGAN-U}$ are the total loss; ${\mathsf{\ell }}_{id}$ denotes identity loss; ${\mathsf{\ell }}_{CGAN}(G,D)$ is the loss of the CGAN; ${\mathsf{\ell }}_{L1}$ denotes the L1 loss; ${\mathsf{\ell }}_{CL}$ is the contrastive learning loss; $E(·)$ denotes the expected operation; $x\in \mathcal{X}$, and the subscript denotes data obtained from a visible light image; $y\in \mathcal{Y}$, and the subscript denotes data obtained from the corresponding real infrared image; y represents the label information obtained from the real infrared image; $D\left(y\right)$ is the probability that the discriminator can accurately determine whether real data are real; $G\left(x\right)$ indicates a target-domain image (i.e., an infrared image) generated based on the source-domain image (i.e., a visible-light image); $D\left(G\right(x\left)\right)$ is the probability that the discriminator can accurately judge whether the generated data are true; finally, ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ are the weights of the LSGAN, L1, and contrastive learning losses in V2IGAN-P, respectively. ${\psi }_1$, ${\psi }_2$, and ${\psi }_3$ are the weights of the LSGAN, identity, and contrastive learning losses in V2IGAN-U, respectively.

The V2IGAN-P algorithm is shown in Algorithm 2.

Algorithm 2 Pseudo code of V2IGAN-P algorithm.

Input: Visible-light image $x\in \mathcal{X}$, Real infrared image $y\in \mathcal{Y}$, Iterations: Iter, the number of iterations for the discriminator is $k_D$, and the number of iterations for the generator is $k_G$; Output: Generated infrared image $G\left(x\right)$;   Initialize G and D model parameters   for all $i=1,2,\cdots ,Iter$ do    for all $j=1,2,\cdots ,K_D$ do     Sampling a sample x from the data distribution $\mathcal{X}$;     Sampling a sample y from the data distribution $\mathcal{Y}$;     Update D by Equation (8)    end for    for all $k=1,2,\cdots ,K_G$ do     Sampling a sample x from the data distribution $\mathcal{X}$;     Generated infrared image $\widehat{y}=G\left(x\right)$;     Calculate L1 loss by Equation (6);     Calculate contrastive learning loss by Equation (5);     Update G by Equation (10);    end for   end for

4. Experimental Results

This section describes the metrics and datasets used in this study and presents the experimental results. All experiments were conducted using an Intel Core i9-10980XE CPU running @ 3 GHz on an Nvidia RTX 3090 GPU.

4.1. Datasets

The FLIR dataset, which was published in 2018, was used for experimental verification in this study. This dataset includes images taken on streets and highways, with infrared wavelengths being long-wave. It should be noted that 9620 pairs of IR–visible-light images from the original FLIR dataset were misaligned; therefore, we performed experiments using the aligned FLIR dataset published by Zhang et al. [75]. The image data were selected and aligned from the original FLIR dataset, totaling 5142 pairs of images; 4118 image pairs were used for training, and 1024 image pairs were used for testing. The images with the black bars were cropped and scaled to the size of 256 × 256. The FLIR dataset was used to verify the authenticity of the infrared image generation of the algorithm presented in this paper in the field of autonomous driving. Some samples of the FLIR dataset are shown in Figure 7a.

The Aerial Visible-to-Infrared Image Dataset (AVIID) [76] is a dataset specifically designed for the task of translating aerial visible light images into infrared images. This dataset was created by researchers from Northwestern Polytechnical University, and it consists of over 3000 pairs of matched visible-light and long-wave infrared images. This dataset provides researchers with a valuable data resource for assessing and improving the performance of algorithms for visible-to-infrared image translation in the aerial domain. For experiments, this study selected the most complex subset of the dataset denoted by AVIID-3. Compared with AVIID-1 and AVIID-2, this dataset contains more types of vehicles and numerous targets of multiple densities, viewpoints, and scales. In addition, AVIID-3 was collected in various scenarios with more complicated backgrounds, including roads, bridges across rivers, parking lots, and streets of residential communities. Therefore, this dataset is more challenging for aerial visible-to-infrared image translation and can be better used to evaluate the performance of different methods. During the experimental process, all images were resized to the dimension of 256 × 256 pixels, with 1024 pairs used for training and 256 pairs for testing. The AVIID dataset was used to verify the authenticity of the infrared image generation of the algorithm presented in this paper in the field of UAV remote sensing. Some samples of the AVIID dataset are shown in Figure 7b.

The IRVI dataset is a dataset designed for infrared-to-visible-light video translation tasks. It contains continuous video clips of traffic and surveillance scenes collected using long-wave infrared cameras, covering low-light conditions and suitable for the fields of autonomous driving and security, with a resolution of 256 × 256. It is aimed at improving the visual signal effect under night or adverse weather conditions. We selected the surveillance scenes from it for the visible-light-to-infrared image translation task. To reduce the redundancy of the target scene images, a total of 1016 pairs of images were used for training, and 255 pairs of images were used for testing. The IRVI dataset was used to verify the performance of the algorithm in the field of surveillance security. Some samples of the IRVI dataset are shown in Figure 7c.

4.2. Implementation Details

The improvements of the V2IGAN-P and V2IGAN-U methods are based on the Pix2Pix and CUT frameworks, respectively. All experiments were conducted using the publicly available source codes and datasets for training and testing in the same experimental environment to provide a fair comparison. It is worth noting that the ThermalGAN algorithm only uses visible light as input. The Adam optimizer was selected, and ${\beta }_1$ and ${\beta }_2$ were set to 0.5 and 0.999, respectively. A total of 200 training epochs were performed to ensure model convergence. The learning rates of the generative and adversarial networks were set to 0.0002 and 0.000002, respectively. In the first 100 epochs, the learning rates were fixed, and then, they decreased linearly to zero. All networks were trained from scratch with random initialization. The experimental parameters for V2IGAN-P were set according to the IR-GAN [47] algorithm as follows: ${\lambda }_1$ = 1, ${\lambda }_2$ = 100, and ${\lambda }_3$ = 10. The experimental parameters for V2IGAN-U were set according to the CVIIT [52] algorithm as follows: ${\psi }_1$ = 1, ${\psi }_2$ = 10, and ${\psi }_3$ = 10.

4.3. Image Quality Evaluation Metrics

Image quality assessment was conducted using a set of established metrics, each of which was used to evaluate different aspects of image fidelity and perceptual quality. The five metrics used in this study included the following:

• (1) Peak signal-to-noise ratio (PSNR) [77]: This is a traditional measure that quantifies the maximum possible signal-to-noise ratio of an image, and it has often been used to assess the quality of reconstructed and compressed images.
  The PSNR is expressed as follows:

  $$PSNR(I,S)=10log{\displaystyle \frac{{(2^n-1)}^2}{MSE(I,S)}},$$

  (11)

  where

  $$MSE(I,S)={\displaystyle \frac{1}{MN}}\sum _{x=1}^M\sum _{y=1}^N{(I(x,y)-S(x,y))}^2,$$

  (12)

  where I is a real IR image, S is a simulated IR image, and M and N are the height and width of the images, respectively.
• (2) Structural Similarity Index Measure (SSIM): This is a perceptual image quality metric that compares local patterns of pixel intensities to evaluate the similarity between two images.
  The SSIM can be defined as

  $$\begin{array}{cc}\hfill SSIM(I,S)& ={\displaystyle \frac{2{\mu }_I{\mu }_S+C_1}{{\mu }_I^2+{\mu }_S^2+C_1}}{\displaystyle \frac{2{\sigma }_I{\sigma }_S+C_2}{{\sigma }_I^2+{\sigma }_S^2+C_2}}{\displaystyle \frac{{\sigma }_{IS}+C_3}{{\sigma }_I{\sigma }_S+C_3}}\hfill \\ \hfill & ={\displaystyle \frac{2{\mu }_I{\mu }_S+C_1}{{\mu }_I^2+{\mu }_S^2+C_1}}{\displaystyle \frac{{\sigma }_{IS}+C_3}{{\sigma }_I^2+{\sigma }_S^2+C_2}},\hfill \end{array}$$

  (13)

  where $C_1={\left(0.01L\right)}^2$, $C_2={\left(0.03L\right)}^2$, and $C_3=C_2/2$ are parameters used to ensure the stability of a partition; ${\mu }_I$ and ${\mu }_S$ are the respective means of I and S; ${\sigma }_I$ and ${\sigma }_S$ are their respective standard deviations; ${\sigma }_{IS}$ is the covariance of I and S; and L is the range of image pixels.
• (3) Multi-Scale Structural Similarity Index Measure (MS-SSIM) [78]: This metric represents an extension of the SSIM that operates on multiple scales, providing a more comprehensive assessment of image quality by capturing both local and global changes.
  The MS-SSIM is defined as

  $$MS-SSIM(I,S)={\left[l_M(I,S)\right]}^{{\alpha }_M}\prod _{j=0}^M{\left[c_j(I,S)\right]}^{{\beta }_j}{\left[s_j(I,S)\right]}^{{\gamma }_j},$$

  (14)

  where $l_M(I,S)$, $c_j(I,S)$, and $s_j(I,S)$ are the brightness, contrast, and structural similarity, respectively; and ${\alpha }_M$, ${\beta }_j$, and ${\gamma }_j$ are weight coefficients for $l_M(I,S)$, $c_j(I,S)$, and $s_j(I,S)$, respectively. To simplify parameter selection, ${\alpha }_j={\beta }_j={\gamma }_j$, ${\sum }_{j=1}^M{\gamma }_j=1$, and $M=5$.
• (4) Learned Perceptual Image Patch Similarity (LPIPS) [79]: This is a recently developed metric that leverages deep learning to measure the perceptual difference between image patches. It has been proposed to provide a closer correlation with human perception of image quality.
  The LPIPS is calculated as

  $$LPIPS(I,S)=\sum _l^L{\displaystyle \frac{1}{H_l{W}_l}}\sum _{H_l,W_l}{[f_l{\left(I\right)}_{H_l,W_l}-f_l{\left(S\right)}_{H_l,W_l}]}^2\ast {\omega }_1.$$

  (15)
  To calculate this, comparative features were obtained from a convolutional neural network-based backbone and pretrained on ImageNet (the AlexNet network model was used in the experiment).
• (5) Fréchet Inception Distance (FID) [80]: This metric is derived from the field of machine learning, and it measures the distance between two statistical distributions of features extracted by the Inception-V3 Network, providing insight into the quality of generated images in a generative model.
  The FID is calculated as

  $$FID(I,S)=\sqrt{\left|\right|{\mu }_I-{\mu }_S{\left|\right|}^2+Tr({\sigma }_I+{\sigma }_S-2\sqrt{{\sigma }_I{\sigma }_S})},$$

  (16)

  where ${\mu }_I$ and ${\mu }_S$ are the respective mean vectors of the real and generative data distributions, with covariance matrices of ${\sigma }_I$ and ${\sigma }_S$, respective; $Tr$ is the trace of the matrix; and $\left|\right|·\left|\right|$ is the binary norm of the vector.

The five selected metrics could collectively provide a multidimensional evaluation of image quality, encompassing both pixel-level accuracy and higher-level perceptual attributes.

4.4. Simulation Evaluation of Infrared Image Generation Quality

Figure 8, Figure 9 and Figure 10 show the examples of infrared images generated on three datasets. The results obtained on the FLIR dataset are presented in Figure 8. It can be seen that the infrared images generated by V2IGAN-P retain more detailed information than those obtained by Pix2Pix, ThermalGAN, and Pix2Pix-MRFFF. In addition, compared with the infrared images generated by ThermalGAN, Pix2Pix-MRFFF, and InfraGAN, the images created by the proposed method exhibit clearer texture information and more realistic gray-scale details. However, all algorithms perform poorly on the FLIR dataset, which could be attributed to the significant variations in the light and shadow in the visible-light images from the FLIR dataset. The results on the AVIID dataset are presented in Figure 9, where it can be seen that the proposed algorithm could generate a clearer outline of the infrared image than the other methods. The infrared images generated by the algorithm in this paper retain more detailed information and distinct contour information for small targets. Integrating the results from Figure 8 and Figure 9, it is evident that the Pix2Pix algorithm experiences severe loss of detail in the generated infrared images, which is attributed to the limited capability of the generative network. The ThermalGAN and Pix2Pix-MRFF algorithms improved the generative network based on the UNet architecture; however, the lack of constraints on the edge structure of the generated images leads to edge distortion in the infrared images. InfraGAN enhanced the constraint on the edge information of the generated images by introducing the SSIM loss, but the discrimination added by the UNet architecture increases the training burden. The results of the IRVI dataset are presented in Figure 10. The V2IGAN-U method generated infrared images with clearer edges and richer textures compared to other unpaired algorithms. However, the infrared images generated by other algorithms had disordered textures, and there were noticeable errors in the expression of infrared features. The algorithm presented in this paper enhanced the generative network’s capabilities by incorporating the visual state space structure, and the introduction of the contrastive learning loss strengthened the network’s perception of fine-grained details in the images, resulting in the best visual outcomes on both datasets. We also conducted a quantitative evaluation of the proposed V2IGAN method to analyze its performance.

The results in

Table 1. Infrared image generation quality evaluation results and training inference time consumption.

FLIR	PSNR ↑	SSIM ↑	MS-SSIM ↑	LPIPS ↓	FID ↓	Training Time (h)	Inference Time (s)
Pix2Pix	22.1529	0.6163	0.5140	0.3312	138.8828	8.94	0.16
ThermalGAN	22.4735	0.6287	0.5475	0.2746	139.2642	6.98	0.17
Pix2Pix-MRFFF	21.8622	0.6158	0.5281	0.3216	126.7535	7.92	0.15
InfraGAN	22.1961	0.6304	0.5411	0.3043	136.1309	25.72	0.17
V2IGAN-P	23.0628	0.6417	0.5726	0.2636	122.9724	66.33	0.19
AVIID	PSNR ↑	SSIM ↑	MS-SSIM ↑	LPIPS ↓	FID ↓	Training Time (h)	Inference Time (s)
Pix2Pix	26.6464	0.7923	0.8143	0.1844	110.9060	2.05	0.16
ThermalGAN	25.1461	0.7520	0.7775	0.1822	99.3957	1.61	0.17
Pix2Pix-MRFFF	23.0939	0.6972	0.6817	0.4106	172.5763	3.27	0.15
InfraGAN	24.9415	0.7603	0.5411	0.1815	96.5861	5.71	0.17
V2IGAN-P	28.0679	0.8376	0.8677	0.1211	71.0606	16.56	0.19
IRVI	PSNR ↑	SSIM ↑	MS-SSIM ↑	LPIPS ↓	FID ↓	Training Time (h)	Inference Time (s)
CycleGAN	12.9192	0.2634	0.1368	0.4268	185.8868	7.76	0.16
CUT	10.6941	0.2603	0.1098	0.5013	175.6123	7.16	0.16
DCLGAN	11.3122	0.2668	0.0297	0.5051	208.3874	11.03	0.16
EMRT	11.0906	0.2635	0.0411	0.5006	188.9970	16.53	0.24
V2IGAN-U	14.9734	0.3876	0.2372	0.3031	136.4583	16.15	0.19

In the table, an upward arrow indicates that the larger the metric, the better the image quality, while a downward arrow indicates that the smaller the metric, the better the image quality. Boldface fonts represent the best values obtained.

indicate that the metrics in the figure are the average results of the test set, and the proposed algorithm achieved the best results in all five metrics. In the table, an upward arrow indicates that the larger the metric, the better the image quality, while a downward arrow indicates that the smaller the metric, the better the image quality. Boldface fonts represent the best values obtained. This suggests that the proposed algorithm is highly competitive in the generation of infrared images. V2IGAN-P achieved a PSNR value of 23.0628 and an SSIM value of 0.6417 on the FLIR dataset. Compared to the results of Pix2Pix and ThermalGAN, the SSIM was improved by approximately 3% and 2%, respectively, and the MS-SSIM was improved by about 6% and 3%, respectively. The larger increase in MS-SSIM compared to SSIM indicates that the algorithm has a good generation effect on the fine-grained details of the image. The LPIPS and FID values were 0.2636 and 122.9724, respectively, indicating that the infrared data generated by the algorithm on the FLIR dataset had a high degree of realism. On the AVIID dataset, V2IGAN-P achieved a PSNR value of 28.0679 and an SSIM value of 0.8376. Compared to the results of Pix2Pix and ThermalGAN, the SSIM was improved by approximately 4% and 8%, respectively, and the MS-SSIM was improved by about 5% and 9%, respectively. The LPIPS and FID values were 0.1211 and 71.0606, respectively, indicating that the infrared data generated by the algorithm on the AVIID dataset had a high degree of realism. On the IRVI dataset, V2IGAN-U achieved a PSNR value of 14.9734 and an SSIM value of 0.3876. Comparing the results of the V2IGAN-U and V2IGAN-P algorithms, it can be observed that the results on unpaired data are relatively poorer than those on paired data. Additionally, the table displays the training and inference times of the model. The training and inference times of the V2IGAN model are acceptable.

Based on the results of both subjective visual assessment and objective metric evaluation, the proposed algorithm demonstrated the best performance on both the FLIR, AVIID, and IRVI datasets.

4.5. Ablation Study

Next, ablation experiments were conducted on the AVIID dataset to verify the effectiveness of combining the visual state space and contrastive learning loss. The ResNet_9 block was used as a baseline generation network, and the PatchGAN model was employed as an adversarial network. The loss functions included the LSGAN and L1 losses. Similarly, the five metrics were used to evaluate the quality of image generation. In this context, “baseline” refers to the Pix2Pix algorithm.

Figure 11 presents a specific example of image generation. From Figure 11c, it can be observed that the image generated by the baseline suffers from severe detail loss and is relatively blurry. As shown in Figure 11d, after incorporating the visual state space attention module, it is noticeable that the details of the generated infrared image are largely preserved. As depicted in Figure 11e, when only the contrastive learning loss is added, it is evident that an infrared image with richer details is generated, but there is a phenomenon of texture loss. As illustrated in Figure 11f, the infrared image generated by our method, compared to the Baseline result, has clearer textures and richer details.

As shown in

Table 2. The results of the ablation experiments conducted on the AVIID dataset.

AVIID	PSNR ↑	SSIM ↑	MS-SSIM ↑	LPIPS ↓	FID ↓
baseline	26.6464	0.7923	0.8143	0.1844	110.9060
+VSS	27.0306	0.8173	0.8489	0.1435	81.7944
+${\mathsf{\ell }}_{CL}$	27.0197	0.7964	0.8356	0.1339	80.3518
V2IGAN	28.0679	0.8376	0.8677	0.1211	71.0606

In the table, an upward arrow indicates that the larger the metric, the better the image quality, while a downward arrow indicates that the smaller the metric, the better the image quality. Boldface fonts represent the best values obtained.

, the addition of the visual state space and contrastive learning loss improved the five indicators to different extents. When both strategies were added simultaneously, the proposed method achieved the best results on the AVIID dataset. This indicated that both the visual state space attention module and the contrastive learning loss were effective. When only the visual state space attention module was added, the results in the table show that the SSIM and MS-SSIM values increased by about 2% and 3%, respectively. When only the contrastive learning loss was added, the SSIM and MS-SSIM values increased by about 0.5% and 2%, respectively. It can be observed that the improvement in the MS-SSIM indicator is greater than the improvement in the SSIM indicator, indicating that the generation effect on multiple scales of the image was enhanced. When both strategies were employed, it was found that compared to the baseline results, the SSIM indicator improved by about 4%, and the MS-SSIM indicator improved by about 5%, indicating that the addition of the visual state space attention module and the contrastive learning loss complement each other, achieving the best indicator results.

5. Discussion

In this study, we observe that the VSS attention module and the contrastive learning loss play important roles. The VSS attention module, which processes information selectively, encourages the generative network to pay more attention to the important features of visible-light images. As shown in Table 1, the addition of the VSS attention module improved all five metrics. Similarly, the contrastive learning loss, through the construction of positive and negative sample pairs, enhanced the feature representation of the images, thus improving the details of the generated images. As shown in Table 1, after the introduction of the contrastive learning loss, the improvement in the MS-SSIM metric was larger than that in the SSIM metric, indicating that contrastive learning effectively improved the fine granularity of the images.

To further demonstrate that the infrared images generated by the proposed method conformed to the distribution of real infrared images, t-distributed Stochastic Neighbor Embedding (t-SNE) [81] technology was utilized to visualize the feature distributions of visible-light images, real infrared images, and the infrared images generated by the V2IGAN method. t-SNE is a type of nonlinear dimensionality reduction technique primarily used for effectively mapping high-dimensional datasets into two or three-dimensional spaces for visualization. It optimizes the embedding in a lower-dimensional space by preserving the relationships of proximity between similar sample points in the higher-dimensional space, making it particularly adept at revealing the local structural features of the data. The greater the overlap in the t-SNE visualization results between two data distributions, the more similar the two datasets are. The feature visualization results on three datasets are shown in Figure 12, Figure 13 and Figure 14, where we use points of different colors and shapes to represent different feature domains. The visualization of features indicates that the feature distribution of the infrared images generated by our method is highly similar to that of the real infrared images, with a large overlap in the areas. The infrared images generated by our method in the fields of autonomous driving, low-altitude space, and surveillance security have similar distributions to the real infrared images, indicating that our method is viable for expanding infrared data.

6. Conclusions

In this work, a novel visible-to-infrared image translation algorithm, named V2IGAN, is proposed based on the VSS attention module and multi-scale feature contrastive learning loss. Firstly, this paper introduces a VSS attention module that strengthens the generative network’s focus on key areas within visible-light images, thereby enhancing both feature extraction and the network’s capacity for image generation. Following that, a multi-scale feature contrastive learning loss function is introduced to align the paired features of images, which refines the fine details of the image generation process. Experimental results on three published datasets demonstrate that the V2IGAN algorithm for converting visible-light to infrared images can produce reliable and high-quality infrared images. Also note that the V2IGAN-P algorithm requires paired data. However, paired training data may not be available for many tasks. This paper further extends the V2IGAN algorithm to generate infrared images without the need for paired visible-light images. However, the generation results from unpaired data are inferior to those from paired data, and the next step will focus on enhancing the algorithm’s performance on unpaired data.