Nighttime Thermal Infrared Image Translation Integrating Visible Images

Abstract

Nighttime Thermal InfraRed (NTIR) image colorization, also known as the translation of NTIR images into Daytime Color Visible (DCV) images, can facilitate human and intelligent system perception of nighttime scenes under weak lighting conditions. End-to-end neural networks have been used to learn the mapping relationship between temperature and color domains, and translate NTIR images with one channel into DCV images with three channels. However, this mapping relationship is an ill-posed problem with multiple solutions without constraints, resulting in blurred edges, color disorder, and semantic errors. To solve this problem, an NTIR2DCV method that includes two steps is proposed: firstly, fuse Nighttime Color Visible (NCV) images with NTIR images based on an Illumination-Aware, Multilevel Decomposition Latent Low-Rank Representation (IA-MDLatLRR) method, which considers the differences in illumination conditions during image fusion and adjusts the fusion strategy of MDLatLRR accordingly to suppress the adverse effects of nighttime lights; secondly, translate the Nighttime Fused (NF) image to DCV image based on HyperDimensional Computing Generative Adversarial Network (HDC-GAN), which ensures feature-level semantic consistency between the source image (NF image) and the translated image (DCV image) without creating semantic label maps. Extensive comparative experiments and the evaluation metrics values show that the proposed algorithms perform better than other State-Of-The-Art (SOTA) image fusion and translation methods, such as FID and KID, which decreased by 14.1 and 18.9, respectively.

1. Introduction

Thermal InfraRed (TIR) cameras, unlike their visible spectrum counterparts, can capture high-quality images in low-light or pitch-dark conditions, and they exhibit superior performance in rainy or foggy scenarios. As a result, they find extensive use in nighttime scene awareness for automated driving systems, drone platforms, and surveillance systems. However, TIR images possess a single monochromatic band, offering less contrast compared to visible images, and often present blurred target boundaries that impede visual interpretation. Daytime Color Visible (DCV) images, on the other hand, are more compatible with human visual perception and facilitate easier environmental interpretation. Additionally, numerous pre-trained models are available for DCV images, which can be leveraged directly for tasks such as target detection [1], tracking [2], pedestrian identification [3], and semantic segmentation [4]. In contrast, there are fewer pre-trained models for TIR images, and their accuracy falls short compared to models designed for DCV images. Given this context, recent years have seen a surge in scholarly interest in translating Nighttime TIR (NTIR) images into DCV images (abbreviated as NTIR2DCV), leveraging machine learning methodologies.

The translation of NTIR2DCV is a mapping from one to three channels. Since it is difficult to build exact matches of NTIR-DCV dataset, most scholars used Unpaired Image Translation (UIT) based on Generative Adversarial Networks (GANs) [5] to train an end-to-end neural network for translating NTIR images into DCV images [6]. However, as the spectral band ranges of TIR and visible images are different, thus, it is difficult to establish a physical or mathematical model describing the relationship between the temperature and color domain. Due to insufficient constraints and the use of unpaired training datasets, existing UIT algorithms cannot solve the ill-posed problem in the translation from temperature to color domain [7]. For example, the DCV images translated by existing UIT algorithms have insufficient color, edge blurring, and semantic error, such as the car, people, road surface, and plants on the road side, as shown in the results in [8].

We think the best approach to dealing with this ill-posed problem of NTIR image translation is to integrate additional information to compensate for the lack of color information in the NTIR image. Urban environments are well-lit at night; therefore, it is easy to capture Nighttime Color Visible (NCV) images, although their quality is relatively low. Integrating color information from NCV image can facilitate the translation of NTIR2DCV. In addition, most current TIR cameras are integrated with CV cameras so that paired NTIR and NCV images can be used for NTIR translation.

Therefore, we propose an NTIR image translation method that integrates NCV images. The NTIR and NCV images are acquired simultaneously, and the NCV image is enhanced because it was acquired under low light conditions. The Enhanced NCV (ENCV) and NTIR images are fused to add color information to the NTIR images, and the fused images are translated into DCV images using machine learning algorithms.

The remainder of this article is organized as follows: Section 2 is a review of related research. Section 3 describes the research methods. Section 4 presents the dataset, experimental methods, and evaluation metrics. Section 5 provides the results, Section 6 presents the discussion, and Section 7 concludes the paper.

2. Related Works

The literature review focuses on nighttime visible image enhancement, TIR and visible image fusion, and GAN-based image translation.

2.1. Enhancement of Low-light Nighttime Visible Images

In contrast to daylight images, NCV images are acquired under low-light conditions and must be enhanced. Existing low-light image enhancement methods typically increase the brightness to improve scene visibility. These methods include Histogram Equalization (HE), Deep Learning (DL) methods, and approaches based on the Retinex theory.

HE methods achieve image brightness and contrast enhancement by stretching the histograms. The principles of related algorithms are simple and easy to implement; therefore, it is widely used [9,10]. However, it does not consider image texture information, and stretching the histogram amplifies the noise in dark areas, decreasing image quality.

Convolutional neural networks have powerful fitting capabilities and can learn complex mapping functions between images acquired under low-light and normal conditions. Therefore, DL-based low-light image enhancement algorithms have attracted widespread attention in recent years [11,12]. The DL-based algorithm provides images with more realistic illumination and natural color than HE algorithms. However, neural networks training depends on the quality of the dataset, and pairs of low-light and normal-light images must be acquired from the same scene. It is difficult to obtain such datasets for outdoor night scenes. Most existing datasets depict indoor scenes and are acquired by turning the light source on and off to control the illumination conditions. In addition, a network trained with these datasets has low generalization ability and is not suitable for enhancement and translation of outdoor images.

Methods based on the Retinex theory assume that the pixel values are the results of the reflectance of objects in the scene and the illumination by ambient light. The enhanced image (reflectance image) should accurately represent the colors of objects in the scene [13,14]. Enhancing a low-light image consists of creating illumination and reflectance images. Since considering the differences in illumination in the scene, the enhancement algorithm based on the Retinex theory can improve the contrast and brightness of the image and represent image details and true colors. Therefore, we use a method based on the Retinex theory to enhance the NCV images and obtain a better performance.

2.2. Thermal InfraRed and Visible Image Fusion

Many studies have investigated the fusion of TIR and visible images. The methods fall into three categories [15]: (1) methods based on Multiscale Transformation (MST), (2) methods based on DL, and (3) methods based on Representation Learning (RL).

The MST-based method includes multiscale forward transformation, the development of fusion rules, and inverse transformation. The Laplace transform [16] and the discrete wavelet transform [17] are typical MST-based methods. Each of these MST-based methods and fusion strategies has its own advantages and disadvantages. For example, the Laplace transform has straightforward implementation and fast operation. However, it does not retain high-frequency details in the fused image. Discrete wavelet transform can fully fuse the feature information of two input images using transform coefficients; therefore, it can obtain a fused image with high signal-to-noise ratio, but its ability to retain image edge information is weak.

DL-based methods can automatically extract meaningful features and fuse the images without the support of default fusion strategies. However, deep neural networks are time consuming and not suitable for real-time computer applications [18,19]. In addition, these methods can only be applied to images with a similar scene content as the training dataset, and their generalization capabilities are limited by the size of the dataset. In addition, obtaining sufficient image quality requires multiple experiments to determine the optimal parameter combination.

RL-based methods include three steps: learning the image representation, designing fusion rules, and performing inverse transformation. The methods based on Sparse Representation (SR) [20] and the methods based on Latent Low-Rank Representation (LatLRR) [21] are typical RL-based methods. An SR model learns an over-complete dictionary and represents the image as the product of the over-complete dictionary and sparse vectors, at first, the sparse vectors are fused, then inverse transformation of the over-complete dictionary is performed to obtain the fused image. In this method, the over-complete dictionary is used to capture the key features of the image and can obtain high fused image quality. However, the learning of over-complete dictionaries is time consuming and inapplicable to real-time applications.

To achieve faster calculations, Li et al. proposed a fusion method based on LatLRR. This method decomposes the source image into basic (low-rank) and salient parts by learning low-rank coefficients and salient coefficient matrices, which is quicker than learning the over-complete dictionary [21]. Subsequently, Li et al. proposed a fusion algorithm based on Multilevel Decomposition LatLRR (MDLatLRR). They believed that the basic part should be further decomposed, i.e., the source image should be decomposed into the low-rank and multiple salient parts, and different methods should be used for fusing the two parts separately, and finally, the fused image was obtained by inverse transformation [22]. MDLatLRR uses multilevel decomposition to extract rich edge information. However, the fusion strategy has limitations because it utilizes an average fusion strategy for the basic part, which reduces the image contrast.

Although many TIR and visible image fusion algorithms exist, few scholars considered the illumination conditions in the algorithm. However, the NCV images have uneven scene illumination, with areas of high brightness around lights, which are difficult to eliminate in fused images [23]. The progressive image fusion network based on illumination-aware (PIAFusion) proposed by Tang et al. uses an illumination perception module to calculate the illumination probability (i.e., the probability of a daytime or nighttime scene) and facilitate the fusion of target features and texture details using an illumination loss function [24]. However, the illumination probability calculation only considers the global illumination difference but not local illumination differences.

Therefore, we propose an improved MDLatLRR algorithm that uses the NCV illumination map to improve image fusion by incorporating local illumination difference into the fusion of the TIR and NCV images to suppress glare in the NCV image.

2.3. GAN-based Thermal InfraRed Image to Visible Image Translation

GANs have been successfully used for generating high-quality images in many fields. Numerous scholars have used them for image translation, including Paired Image Translation (PIT) [25,26] and UIT [27,28]. Most studies used the UIT algorithm for NTIR to DCV image translation due to the difficulty in obtaining a sufficient number of pixel-level registered NTIR-DCV datasets.

Luo et al. was the first author who used the UIT method for the translation of NTIR2DCV using DlamGAN [6]. Two semantic segmentation modules were added to CycleGAN to extract label maps of the source and generated images. This model constrains the semantic consistency in the translation process using a semantic loss function. However, this method requires accurate semantic segmentation of the real and generated NTIR and DCV images in the training process. If the semantic segmentation error is large, anomalous colors and textures will occur in the translated image.

The PearlGAN model [8] uses a top-down attention module to extract multiscale contextual information from the image and improve the semantic consistency between the source and generated images. It uses an attention loss function to reduce the overall and local semantic ambiguity of the translated image. However, experimental results have shown low performance of the attention module. The generated images have many erroneous semantic translation results, such as street lights translated into the sky or merged buildings and trees. Li et al. used a video streaming dataset and evaluated the spatiotemporal consistency between consecutive image frames to maintain semantic consistency during the translation [29]. However, the experimental results showed low performance and incorrect semantic translated results, such as vehicles being identified as roads.

Therefore, the most significant problem in UIT is the semantic inconsistency of the images before and after translation.

In summary, since existing image fusion and UIT algorithms suffer from various problems, this paper proposes a novel method of NTIR2DCV. A dual-lens camera is used to acquire NTIR and NCV images. The NCV image is enhanced based on the Retinex theory. The MDLatLRR algorithm is improved to optimize the fusion of the TIR and ENCV images. Since Hyperdimensional Computing (HDC) can express the semantic and style information of images and calculate them separately [30], we introduced HDC into UIT and proposed HDC-GAN to translate the fused nighttime image into a daytime image while retaining the semantic information, improving the semantic consistency between the two image types.

3. Research Methods

Figure 1 illustrates our NTIR2DCV method, which consists of three parts: (1) The illumination map of the low-light NCV images is created based on the Retinex theory and is used to enhance the NCV and obtain the ENCV image. (2) The illumination map is used in the illumination-aware MDLatLRR (IA-MDLatLRR) to fuse the ENCV and NTIR images. The IA-MDLatLRR calculates the illumination weight based on the illumination map and creates adaptive fusion rules based on basic and salient parts under different illumination conditions to obtain Nighttime Fused (NF) images unaffected by glare. (3) The HDC-GAN algorithm is proposed to improve semantic consistency during image translation. HDC-based loss functions are incorporated in HDC-GAN to ensure the semantics are consistent. We describe the details of each part below.

3.1. Illumination Map Calculation and NCV Image Enhancement

According to the Retinex theory, the low-light image $L$ consists of the illumination-enhanced image $R$ and the illumination map $T$[14]. $L$ is expressed as follows:

$$L=R\circ T$$

(1)

where the operator $\circ$ represents the matrix dot multiplication.

As shown in Figure 2, the original Figure 2a can be expressed as the product of the illumination-enhanced Figure 2b and the illumination map Figure 2c. According to Equation (1), the enhanced image can be obtained by calculating $R=L/T$. Therefore, the core step of the image enhancement method based on the Retinex theory is the calculation of the illumination map $T$.

The illumination map represents the scene illumination. The calculation based on the Retinex theory has two steps: (1) estimation of the initial illumination map (Figure 2d) and (2) refinement of the initial illumination map to obtain the final illumination map using the constraint conditions from the initial illumination map. Two constraints are added. The first is image smoothing based on texture details because the illumination in adjacent areas with similar textures is similar. The second is to retain the structure of the initial illumination map. The purpose of the illumination map optimization is to enhance the texture of $R$ by smoothing $T$.

We use the Max-RGB method, to estimate the initial illumination map [31]. It is assumed in the Max-RGB method that the three channels of the low-light image $L$ have the same illumination conditions in a nighttime image. The initial estimate of the pixel value x of the illumination map is the maximum pixel value of the three channels R, G, and B:

$$\hat{\mathrm{T}}\left(x\right)=\underset{c\in \left\{R,G,B\right\}}{max}{\mathrm{L}}^c\left(x\right)$$

(2)

The illumination map optimization algorithm proposed in [14] is simple, efficient, and does not introduce noise. We use this algorithm to optimize the illumination map. This algorithm uses the following objective function to minimize the values of the two constraints of the initial illumination map optimization:

$$\underset{\mathrm{T}}{min}\left(\parallel \hat{\mathrm{T}}-\mathrm{T}{\parallel }_F^2+\alpha \parallel {\mathrm{W}}_h\circ {\nabla }_h\mathrm{T}+{ \mathrm{W}}_v\circ {\nabla }_v\mathrm{T}{\parallel }_{L1}\right)$$

(3)

where $\parallel \cdot {\parallel }_F$ and $\parallel \cdot {\parallel }_{L1}$ are the Frobenius norm and the L1 norm, respectively, and $\alpha$ is the coefficient of the balanced two terms. ${\mathrm{W}}_h$ and ${\mathrm{W}}_v$ are the weight matrices in the horizontal and vertical directions, and ${\nabla }_h$ and ${\nabla }_v$ are the first-order derivative filters in the horizontal and vertical directions, respectively. In the objective function (3), the smaller the first term, the more similar the structure of the initial illumination map $\hat{\mathrm{T}}$ and the final illumination map $\mathrm{T}$. The smaller the second term, the smoother the texture and edges of the final illumination map $\mathrm{T}$.

The key to optimizing the initial illumination map in the objective function (3) is the design of $\mathrm{W}$, i.e., the calculation of the Relative Total Variance (RTV) to calculate $\mathrm{W}$, we use the algorithm in [32] as follows:

$$\begin{array}{c}{\mathrm{W}}_h\left(x\right)={\displaystyle {\displaystyle \sum }_{y\in \mathsf{\Omega }\left(x\right)}}\frac{G_{\sigma }\left(x,y\right)}{\left|{{\displaystyle \sum }}_{y\in \mathsf{\Omega }\left(x\right)}{G}_{\sigma }\left(x,y\right){\nabla }_h\hat{\mathrm{T}}\left(y\right)\right| + ϵ}\\ {\mathrm{W}}_v\left(x\right)={\displaystyle {\displaystyle \sum }_{y\in \mathsf{\Omega }\left(x\right)}}\frac{G_{\sigma }\left(x,y\right)}{\left|{{\displaystyle \sum }}_{y\in \mathsf{\Omega }\left(x\right)}{G}_{\sigma }\left(x,y\right){\nabla }_v\hat{\mathrm{T}}\left(y\right)\right| + ϵ}\end{array}$$

(4)

where $\mathsf{\Omega }\left(x\right)$ is the area centered on pixel $x$; $y$ is the position index in the area; $ϵ$ is a very small constant to prevent the denominator from being 0; $G_{\sigma }\left(x,y\right)$ is a Gaussian kernel function with standard deviation $\sigma$ that can be expressed as:

$$G_{\sigma }\left(x,y\right)=\exp \left(-\frac{\mathrm{dist}\left(x,y\right)}{2{\sigma }^2}\right)$$

(5)

where the function $\mathrm{dist}\left(x,y\right)$ measures the Euclidean distance between locations $x$ and $y$.

After obtaining $\mathrm{W}$, we can solve the optimization problem described in Equation (3) to obtain the illumination map $\mathrm{T}$ and the ENCV image according to Equation (1).

3.2. NTIR and NCV Image Fusion Based on IA-MDLatLRR

The MDLatLRR method decomposes the image into basic and salient parts. The salient part contains edge, texture, and other detailed information, and the basic part contains intensity information. In this method, an average fusion strategy is employed for the basic part, which consequently moderates the brightness and contrast of the resultant fused image. We use the illumination maps to improve the fusion strategy of MDLatLRR and propose an NTIR and NCV image fusion algorithm named IA-MDLatLRR, whose diagram is shown in Figure 3.

First, the illumination map of the NCV image is obtained based on the Retinex theory, and the ENCV image is derived using Equation (1). Then, the ENCV image is decomposed into the YCbCr channels, and the Cb and Cr channels containing color information are merged with the grayscale NF image in the last step to obtain the final color NF image. The Y channel and TIR image (denoted as $I_1$ and $I_2$, respectively) without color information are used as the inputs of the MDLatLRR algorithm. $I_1$ and $I_2$ are decomposed $r$ times by MDLatLRR to obtain r pairs of matrices of the salient part ($V_{dk}^{1:r}$), an image pair of the basic part ($I_{bk}^r$), and $k\in \left\{1,2\right\}$. The two parts are each fused to obtain fused salient images $I_{df}^{1:r}$ and a fused basic image $I_{bf}$. The two are combined to obtain the final fused image.

Comparing Figure 4a,b shows that the ENCV image is affected by high illumination conditions, resulting in glare and other phenomena not conducive to visual interpretation, whereas the TIR image is not. Therefore, the weight of the TIR image is increased during the fusion in areas with glare. The ENCV shows more details and has more realistic colors than the TIR image in areas with better illumination; thus, the weight of the ENCV image is increased in these areas.

We use illumination weights to improve the fusion of the basic part and balance the impact of the illumination conditions using the following fusion expression:

$$I_{bf}\left(x,y\right)=w_b\circ I_{b1}^r\left(x,y\right)+\left(1-w_b\right)\circ I_{b2}^r\left(x,y\right)$$

(6)

where $\left(x,y\right)$ are the pixel coordinates corresponding to the basic part ($I_{b1}^r$ and $I_{b2}^r$) and the fused basic part ($I_{bf}$). The illumination weight $w_b$ is calculated as follows:

$$w_b=Norm\left(\frac{1}{T_{ENCV}}\right)$$

(7)

where $Norm$ denotes the normalization operation, and $T_{ENCV}$ is the illumination map (Figure 4c) of the ENCV image (Figure 4a) derived from Equations (2) and (3). Since a reciprocal operation is performed on the illumination map during the enhancement in Equation (1), we also perform the reciprocal operation in Equation (7) to calculate the illumination weight $w_b$ (Figure 4d).

The salient part retains more structural information and salient features than the basic part. We use the nuclear norm and illumination weight to calculate the fusion weight of the salient part’s matrix $V_{dk}^{1:r}$ and fuse it column by column. The operation is shown in Figure 5.

$V_{d1}^i$ and $V_{d2}^i$ are the matrices of the salient part obtained by decomposing the Y channel of the ENCV ($I_1$) and TIR ($I_2$) images for the $i$-th time by MDLatLRR, respectively; $V_{d1}^{i,j}$ and $V_{d2}^{i,j}$ are the column vectors corresponding to the $j$-th column, respectively; $V_{df}^i$ is the fusion matric of the final salient part. The calculation formulas of the nuclear norm weights ($w{k}_{d1}^{i,j}$ and $w{k}_{d2}^{i,j}$) corresponding to $V_{d1}^{i,j}$ and $V_{d2}^{i,j}$ are as follows:

$$\begin{array}{c}w{k}_{dk}^{i,j}=\frac{w{k}_{dk}^{\widehat{i},j}}{{{\displaystyle \sum }}_{k=1}^{K}w{k}_{dk}^{\widehat{i},j}}\end{array} ; w{k}_{dk}^{\widehat{i},j}={\Vert re\left(V_{dk}^{i,j}\right)\Vert }_{kernel}\left(k=1, 2\right)$$

(8)

where $re(\cdot )$ represents the operator that transforms the column vector $V_{dk}^{i,j}$ into image blocks; $\parallel \cdot {\parallel }_{kernel}$ represents the calculation of the nuclear norm. The nuclear norm calculates the sum of singular values, preserving the two-dimensional information of the source image. The weight of the nuclear norm is larger when an input image patch contains more texture and structural information.

The calculation formulas of the salient part’s illumination weights $w{i}_{d1}^{i,j}$ and $w{i}_{d2}^{i,j}$ corresponding to $V_{d1}^{i,j}$ and $V_{d2}^{i,j}$ are as follows:

$$w{i}_{d1}^{i,j}=AVER\left(V{i}_d^{i,j}\right); w{i}_{d2}^{i,j}=1-w{i}_{d1}^{i,j}$$

(9)

where $AVER$ is the averaging operation of the column vector. The calculation formula of the illumination matrix $V{i}_d$ is as follows:

$$V{i}_d=P\left(w_b\right)$$

(10)

where $P(\cdot )$ is an operator that changes the illumination weight $w_b$ into the illumination matrix $V{i}_d$. This operation has two steps. The first step is a calculation based on a sliding window with a size of $n\ast n$ and a step of size $s$. The image blocks corresponding to each window are rearranged into a column vector. In the second step, all rearranged column vectors are transformed into a matrix $V{i}_d$.

After calculating the nuclear norm and illumination weight, the column vector $V_{df}^{i,j}$ corresponding to the $j$-th column of the fusion matrix $V_{df}^i$ is obtained by weighting it as follows:

$$V_{df}^{i,j}={\displaystyle \sum }_{k=1}^K\left(\lambda \times w{i}_{dk}^{i,j}+\left(1-\lambda \right)\times w{k}_{dk}^{i,j}\right)\times V_{dk}^{i,j } \left(k=1, 2\right)$$

(11)

where $\lambda$ is the coefficient to balance the illumination weight $w{i}_{dk}^{i,j}$ and the nuclear norm weight $w{k}_{dk}^{i,j}$.

After obtaining the salient part’s fusion matrix $V_{df}^i$, it is reconstructed into the salient image $I_{df}^i$:

$$I_{df}^i=R\left(V_{df}^i\right),i=1,2,\cdots ,r$$

(12)

where $R(\cdot )$ is the reconstruction operator that rearranges each column of $V_d$ into image blocks and averages the overlapping pixels between the image blocks.

After obtaining the fused salient images $I_{df}^{1:r}$ and the basic image $I_{bf}$, they are combined to obtain the final fused image:

$$I_f\left(x,y\right)=I_{bf}\left(x,y\right)+{\displaystyle \sum }_{i=1}^r{I}_{df}^i\left(x,y\right)$$

(13)

3.3. Nighttime to Daytime Image Translation Based on HDC-GAN

It is assumed that an image $x$ from the source domain $X$ (nighttime scene) contains a dark blue sky and bright street lights. Without losing generality, we assume that the image feature corresponding to $x$ is a hypervector $v_x$, which can be expressed as the semantic symbols of the sky and street lights and their style symbols:

$$v_x=c\otimes c_x+p\otimes p_x$$

(14)

where $c$ and $p$ are semantic symbols representing the sky and street lights, respectively, $c_x$ and $p_x$ are style symbols representing the dark blue color of the sky and the brightness of the street lights, respectively; $\otimes$ is the HDC binding/unbinding operator [33], representing the relationship between the semantic symbols c and p and their styles $c_x$ and $p_x$.

Since HDC can express the hypervectors, we use it to express the semantic and style information of the images. Neubert found that the hypervector of the image features could be expressed using a Locality-Sensitive Hash (LSH) table [33].

The goal of image translation is to unbind the semantic symbols (which are expressed as hypervectors in HDC) from the style symbols in the source domain $X$ and bind them with the style symbols in the target domain $Y$. This process requires training the mapping hypervector $u_{X\leftrightarrow Y}$ between the source domain $X$ and the target domain $Y$. If the hypervector expressing the image features of the target image $y$ is $v_y$, the mapping relationship between it and the image feature hypervector $v_x$ corresponding to the source image $x$ can be expressed as:

$$v_y\approx v_x\otimes u_{X\leftrightarrow Y}$$

(15)

where:

$$\begin{array}{c} u_{X\leftrightarrow Y}=\left(c_x\otimes c_y+p_x\otimes p_y\right)\end{array}$$

(16)

Uncorrelated hypervectors in the HDC hyperspace are orthogonal; therefore, the noise generated by binding and unbinding incorrect styles (such as $c\otimes p_y$) in Equation (15) is negligible [34]. Since the binding/unbinding operation is reversible (for example, $c\otimes c_x\otimes c_x=c$), $c$ and $p$ are unbound from the source domain style symbols $c_x$ and $p_x$ and bound with the target domain style symbols $c_y$ and $p_y$ in Equation (15). This strategy ensures the consistency of the semantic information in the style transfer. It is worth noting that $u_{X\leftrightarrow Y}$ is equivalent to $u_{Y\leftrightarrow X}$, i.e., a single mapping can be used to perform forward and reverse mapping between the target and source domains: $v_y\otimes u_{X\leftrightarrow Y}\approx v_x$.

We propose the HDC-GAN model to prevent semantic errors during UIT in the translation from the NF to the DCV image, as shown in Figure 6. HDC-GAN has three main network components: a generator $G$, a discriminator $D_Y$, and a mapping network $F$. It has two loss functions: a hypervector adversarial loss (GAN loss for short) and a hypervector cycle consistency loss (HDC loss for short).

HDC-GAN uses unpaired images as input and trains the generator $G$ to translate the source image $x$ from the source domain $X$ to the target domain $Y$ to obtain the translated image $G\left(x\right)$. The discriminator $D_Y$ is trained to distinguish whether the hypervectors $v_y$ and $v_{G\left(x\right)}$ corresponding to the target image $y$ and the translated image $G\left(x\right)$ belong to the target domain $Y$. We use the pre-trained VGG19 network [35] to extract the features of image $x$, $y,$ and $G\left(x\right)$. Then, we use the LSH method to project these image features to the hypervectors denoted as $v_x$, $v_y$, and $v_{G\left(x\right)}$.

Furthermore, we train the mapping network $F$ to learn the hypervector mapping $u_{X\leftrightarrow Y}$ between the source and target domains. Specifically, we input $v_x$ to $F$ and train $F$ to output $u_{X\leftrightarrow Y}$, i.e., $u_{X\leftrightarrow Y}=F\left(v_x\right)$. The reversible $u_{X\leftrightarrow Y}$ performs two operations simultaneously: unbinding the style of the source domain (or target domain) and binding the style of the target domain (or source domain). We bind $u_{X\leftrightarrow Y}$ to the translated hypervector $v_{G\left(x\right)}$, and the resulting inverted translated hypervector, which is mapped to the source domain, is denoted as $v_{G\left(x\right)\to x}$. We bind $u_{X\leftrightarrow Y}$ to the source hypervector $v_x$, and the resulting hypervector mapped to the target domain is denoted as $v_{x\to y}$:

$$\begin{array}{cc}{v}_{G\left(x\right)\to x}=v_{G\left(x\right)}\otimes u_{X\leftrightarrow Y}& \\ v_{x\to y}=v_x\otimes u_{X\leftrightarrow Y}& \end{array}$$

(17)

The GAN loss is calculated to train $G$, $D_Y,$ and $F$ to learn different styles of the target and source domains. We input the hypervectors $v_y$, $v_{x\to y}$, and $v_{G\left(x\right)}$ belonging to the target domain to the discriminator $D_Y$ to calculate the GAN loss:

$$\begin{array}{cc}{ℒ}_{GAN}\left(G,D_Y,F,X,Y\right)& ={\mathbb{E}}_{y\sim p_Y\left(y\right)}\left[\log D_Y\left(v_y\right)\right]\\ & +{\mathbb{E}}_{x\sim p_X\left(x\right)}[\log \left(1-D_Y\left(v_{G\left(x\right)}\right)\right]\\ & +{\mathbb{E}}_{x\sim p_X\left(x\right)}[\log \left(1-D_Y\left(v_x\otimes F\left(v_x\right)\right)\right]\end{array}$$

(18)

Although the adversarial training method can generate images that match the target style, images with semantic errors may correspond to the target style when only the GAN loss is used. Therefore, we need an additional loss to constrain the generator and preserve the semantic information of the source domain while transforming the style. Thus, we propose the HDC loss. When the translated hypervector is mapped back from the target domain to the source domain, it is expected that the obtained hypervector $v_{G\left(x\right)\to x}$ is the same as the original source hypervector:

$${ℒ}_{HDC}\left(G,X\right)={\mathbb{E}}_{x\sim p_X\left(x\right)}dist\left(v_x,v_{G\left(x\right)\to x}\right)$$

(19)

where $dist$ is the cosine distance between the hypervectors. We minimize the cosine distance between $v_{G\left(x\right)\to x}$ and $v_x$ to ensure that the inverted translated hypervector has the same semantics and styles as the source hypervector.

The total loss function of the HDC-GAN consists of the GAN loss and the HDC loss and is expressed as:

$$ℒ\left(G,D_Y,F,X,Y\right)={ℒ}_{GAN}\left(G,D_Y,F,X,Y\right)+\lambda {ℒ}_{HDC}\left(G,X\right)$$

(20)

where λ is a parameter that adjusts the relative proportion of the two loss functions.

4. Experimental Methods and Evaluation Metrics

4.1. Datasets

Since this research focuses on the translation of NTIR images fused with NCV images, it has strict requirements for the dataset for training the neural network. Most of the images in the TIR and CV datasets KAIST and FLIR were acquired during the daytime. The number of visible images taken at night is small and not suitable for this study. Therefore, we used the DJI ZENMUSE XT2 camera to collect a dataset including 268 pairs of NTIR and NCV images in urban scene. The NTIR images were acquired in the black heat mode, which means that the darker the pixels in the NTIR image, the higher the temperature value. All data were used for training and testing in the fusion experiment. In the translation experiment, 80% of the images were selected as the training set. The remaining 20% of the images were used as the test set to calculate quantitative indicators and compare the translated images obtained by different methods.

The visible camera of the XT2 was used to collect about 300 images of daytime scenes. The original resolution of the NCV images and DCV images was 4000 × 3000, and the original resolution of the TIR images was 640 × 512. For the convenience of fusion and translation, we registered the NCV and NTIR images and down sampled NCV to 640 × 512.

4.2. Experimental Methods

The hardware configuration of the desktop computer used in the image fusion experiment was as follows: The CPU was Intel i7-11700, 16 G 3000 MHz memory was used, and the code was written using MATLAB R2022b. The image translation experiment used a Dell PowerEdge T640 tower server with an Intel Xeon Silver 4116 CPU, an Nvidia 1080Ti GPU, and 128 G 3200 MHz memory.

In the image fusion experiment, for IA-MDLatLRR, we set the decomposition number $r$ of the source image to 2. We introduced illumination weights to fuse the basic part obtained from image decomposition. The main configuration parameters were the same as in [22]; in the fusion of the salient part, we set the coefficient $\lambda$ to 0.5 to balance the illumination weight and the nuclear norm weight.

We conducted three image translation experiments to compare the translations from TIR to DCV, NCV to DCV, and NF to DCV. Before the translation, the input images were scaled from 640 × 512 to 256 × 256. We chose the same generator as the UNSB method in [36] because it is one of state-of-the-art DL methods.

The discriminator and mapping network used fully convolutional networks. The discriminator used a three-layer fully convolutional network with a 1 × 1 convolution filter. Its first layer input dimension (i.e., the hypervector dimension) was 4096, the output dimensions of the first two layers were 1024, and the output dimension of the last layer was 1. The mapping network used a two-layer fully convolutional network, and the input and output dimensions were 4096.

An Adam optimizer was used (${\beta }_1=0.5$, ${\beta }_2=0.999$) [37] with a batch size of 1, a learning rate of the generator and mapping network of 0.0001, and a learning rate of the discriminator of 0.0004. Additionally, $\lambda$ was 10 in the total loss function (Equation (20)).

We compared the results of three fusion methods with our proposed image fusion method: MDLatLRR [22], DenseFuse [38], and SwinFusion [39]. Four translation methods were compared with our HDC-GAN: CycleGAN [27], CUT [40], QS-attn [28], and UNSB [36], whose training duration is 6 h, 12 h, 20 h, and 28 h, respectively, while training our HDC-GAN requires 16 h. In addition, the training epoch of all neural networks is set to 200.

4.3. Evaluation Metrics

We used four indicator types to evaluate the quality of the fused images: information entropy, image feature, correlation, and human perception-based indicators. The indicators based on information entropy included ENtropy (EN) [41], Mutual Information (MI) [42], Peak Signal-to-Noise Ratio (PSNR) [43], and an indicator based on edge information ($Q_{abf}$) [44]. The indicators based on image features included the Average Gradient (AG) [45] and Standard Deviation (SD) [46]. Indicators based on correlation included the Correlation Coefficient (CC) [47] and the Sum of the Correlation Differences (SCD) [48]. The indicator based on human perception was Visual Information Fidelity (VIF) [49]. The higher the value of the nine image fusion indicators, the better the model performance.

We chose the widely used Frechet Inception Distance (FID) [50] and Kernel Inception Distance (KID) [51] to measure the similarity between the real and generated images and assess image translation performance. The lower the value of the two image translation indicators, the better the model performance.

5. Experimental Results

5.1. Image Fusion Experiment

We conducted fusion experiments with four methods, including the proposed method. Due to space limitations, Figure 7 only shows the fusion results of six images. Figure 7a is the NTIR image, Figure 7b is the Y channel of the ENCV image, and Figure 7c–f show the fusion results of MDLatLRR, DenseFuse, SwinFusion, and the proposed IA-MDLatLRR.

The results of the nine evaluation indicators are listed in

Table 1. Objective evaluation results of image fusion comparison experiment. Blue numbers represent the highest-ranking method, and green numbers represent the second-ranked method.

Method	EN	MI	PSNR	${\mathit{Q}}_{\mathit{a}\mathit{b}\mathit{f}}$	AG	SD	CC	SCD	VIF
DenseFuse [38]	6.5496	2.1373	65.3443	0.3964	2.9437	24.8076	0.6523	1.4365	0.6340
SwinFusion [39]	6.8680	2.4085	62.8285	0.5758	5.3090	34.4778	0.5883	1.4347	0.6933
MDLatLRR [22]	6.8430	1.8459	63.3278	0.4907	4.2988	30.1543	0.5215	1.1690	0.6363
IA-MDLatLRR	7.0048	2.7012	65.6855	0.6281	4.8661	34.3022	0.6769	1.5607	0.7619

. In terms of entropy-based metrics, MDLatLRR surpasses the second-ranked method by margins of 0.14, 0.29, 0.34, and 0.05 for EN, MI, PSNR, and $Q_{abf}$, respectively. This indicates that images fused via the IA-MDLatLRR technique contain a more substantial information. When considering correlation-based measures, MDLatLRR’s CC and SCD values outperform those of its closest competitor by 0.02 and 0.12, respectively. This implies a greater degree of similarity between the source images and those fused by the IA-MDLatLRR method. From a human perception-based standpoint, MDLatLRR’s VIF value exceeds that of the second-best method by 0.07. This suggests that images produced by the IA-MDLatLRR method exhibit better visual fidelity.

Despite its strengths, IA-MDLatLRR exhibits a slightly inferior performance to SwinFusion on image feature-based metrics (AG and SD). This is primarily due to the fact that SwinFusion’s loss function yields more texture information. Thermal InfraRed (TIR) images of nocturnal scenes are rich in gradients and textures. Our proposed method takes into account information from both visible and TIR images, which leads to lower indicator values compared to SwinFusion. However, SwinFusion is a neural network method that requires extensive training time, and its generalization ability is affected by the dataset. It is less convenient than the proposed method for practical applications. Moreover, the VIF and SD metrics for IA-MDLatLRR surpass those of MDLatLRR, indicating that our improvements resulted in better visual performance and higher contrast.

The fused images obtained by the DL-based algorithm (Figure 7d,e) retained most glare in the original visible image (Figure 7b). The results of the proposed method (Figure 7f) were less affected by illumination. For the convenience of comparison, we display the fused images of different algorithms in the fourth row of Figure 7 and Figure 8. The fused image obtained by IA-MDLatLRR with the illumination weight was closer to the TIR image. The texture is richer, and objects can be better distinguished from the background area than in the images derived from the other algorithms.

5.2. Image Translation Experiments

The results of the image translation experiment are listed in

Table 2. Objective evaluation results of image translation comparison experiment. Blue numbers represent the highest-ranking method, and green numbers represent the second-ranked method.

Method	NF- > DCV		NTIR- > DCV		NCV- > DCV
	FID	$\mathbf{KID} \mathbf{(}\mathbf{*}{\mathbf{10}}^{\mathbf{3}}\mathbf{)}$	FID	$\mathbf{KID} \mathbf{(}\mathbf{*}{\mathbf{10}}^{\mathbf{3}}\mathbf{)}$	FID	KID ($\mathbf{*}{\mathbf{10}}^{\mathbf{3}}\mathbf{)}$
CycleGAN [27]	109.6172	38.1537	121.8253	47.5827	130.7170	55.6975
CUT [40]	127.4195	58.4600	143.4373	78.3309	180.2604	101.1322
QS-attn [28]	123.4470	50.4946	132.0227	62.0031	133.2309	59.0322
UNSB [36]	108.6397	36.7278	120.5463	45.8972	125.1293	49.5961
HDC-GAN	94.5200	17.8051	108.9323	36.1451	111.5395	43.4754

. The NF to DCV translation performed the best, followed by the NTIR to DCV and NCV to DCV translations. The image derived from the NF to DCV translation was more realistic than that from the NTIR to DCV translation, demonstrating the necessity of combining nighttime color information and temperature information. The image derived from the NTIR to DCV translation was more realistic than that from the NCV to DCV translation. The reason is that a thermal sensor captures more texture information than a visible sensor in nighttime scenes. Therefore, the results obtained from the three sets of experiments are consistent with our expectations.

The proposed HDC-GAN outperformed the other algorithms in the three sets of experiments. In Table 2, in NF- > DCV process, the HDC-GAN method demonstrated superior performance with FID and KID scores being 14.1 and 18.9 lower than those of the second-ranked UNSB. In the NTIR- > DCV process, HDC-GAN showed significant improvements, with reductions in FID and KID values by 11.6 and 9.7, respectively when compared with UNSB. Furthermore, in the NCV- > DCV process, HDC-GAN performs much better than UNSB, resulting in FID and KID scores lower than UNSB with a value of 13.6 and 6.1, respectively.

(a)

NCV to DCV

Figure 9 shows the results of the NCV to DCV translation. All the UIT algorithms were able to translate unenhanced low-light NCV images into bright daytime images. Figure 10 shows the magnification of some images derived from CycleGAN, UNSB, and HDC-GAN. The other UIT algorithms exhibited many semantic errors, such as numerous irregular spots in the sky, and unrealistic shadows on the road. The buildings, vehicles, and trees were distorted and difficult to recognize. In contrast, the image translated by the HDC-GAN was darker and showed more semantic consistency with the NCV image.

(b)

NTIR to DCV

Figure 11 shows the results of the NTIR to DCV translation. The first column in Figure 11a shows the NTIR image. The other algorithms in Figure 11b–e translated the NTIR image into daytime images with good color effects. However, the inconsistency in the semantic information was greater than in the NCV to DCV translation. The main reason is that the UIT algorithms cannot solve the color ambiguity when translating temperature to brightness information due to the lack of color in the temperature domain. The NTIR image was randomly colored, resulting in distortions in the color domain and errors in semantic information. The image translated by the HDC-GAN algorithm (Figure 11f) exhibited good semantic consistency but did not provide the expected color translation. The main reason is that the NTIR image has no color information. Although the target image contains color information, the HDC emphasizes semantic consistency, and it is still difficult to incorporate color information into the learning process.

(c)

NF to DCV

Figure 12 shows the results of the NF to DCV translation. The first column of the images shown in Figure 12a is the fused ENCV and NTIR images, i.e., the NF image. Figure 12b–e show the translation results derived from the CycleGAN, CUT, QS-attn, and UNSB algorithms, respectively. Figure 12f is the translation result of HDC-GAN. The HDC-GAN outperformed the other algorithms in terms of color and semantics.

Since the NF image has good color and texture, the images translated by all algorithms are more realistic than those in Figure 9 and Figure 11. Although the semantic ambiguity in the images translated by other UIT methods is less pronounced than that in Figure 9 and Figure 11, these algorithms produce more semantic errors than HDC-GAN.

Figure 13 shows the fused images derived from CycleGAN, UNSB, and HDC-GAN in the first, third, and sixth rows of Figure 12. In the first row, the image obtained by HDC-GAN indicates that the trees in the upper left area, the vehicles on the right, and the road below show semantic consistency. However, the images obtained from the other two UIT algorithms exhibit anomalous texture areas in the trees, shadows on the road, and some distorted vehicles. These problems exist in all of the translation results of the CycleGAN and UNSB algorithms.

In the second row, the image obtained by HDC-GAN shows good detail in the background building and accurate semantics. In contrast, the texture in the images obtained by UNSB is blurred, and the image obtained by CycleGAN has noise in the sky and semantic errors in the building, which is darker than in the NF image. The third row shows that in the image obtained by HDC-GAN, the semantics of the speed bump (at the bottom of the picture) and the white line (on the left side of the vehicle on the right side of the picture) remain consistent, whereas the image obtained by UNSB exhibits significant distortions and semantic errors. In the image obtained by CycleGAN, the right side of the speed bump is translated to a road, the texture of the white line is blurred, and its color becomes yellow.

6. Discussion

6.1. Eliminating the Influence of Lights in Nighttime Images

Although integrating NCV images significantly enhanced the translation of NTIR images, the issue of glare from lights persisted during the translation to DCV images. To address this challenge, we employed illumination weights in IA-MDLatLRR and assigned greater weight to NTIR images in areas with strong illumination. This strategy mitigated the impact of glare.

However, using larger weights enhances texture information but does not compensate for the absence of color information in NTIR images. Consequently, in the NF image, the area affected by glare exhibits high brightness and appears white rather than the actual color. In future research, we will consider learning color information from images captured at varying angles, as the illumination in nighttime sequence images (such as video streams) varies depending on the viewing angle.

Additionally, our current method for calculating illumination weights is relatively straightforward. In future research, we plan to implement illumination weights in the neural network similarly to PIAFusion [24] and utilize an illumination-aware loss function for adaptive fusion of target features and texture details. This approach should enhance the fusion results of images affected by glare.

6.2. Integrating NCV to NTIR Translation

The translation results from NCV to DCV shown in Figure 9 seem to be satisfactory without NTIR images. However, NCV images suffer from a lack of texture information, especially in dimly lit areas, making some scenes or targets difficult to identify. In such scenarios, NTIR images provide more texture details that facilitate target recognition. Therefore, by fusing NCV and NTIR images, we can achieve more robust scene and target information, paving the way for further research on target recognition or scene segmentation.

6.3. HDC-GAN Algorithm Used to Process Remote Sensing Images

In order to verify the applicability of this method in the field of UAV remote sensing, we used a dataset collected by a DJI M300 drone and a DJI ZENMUSE XT2 camera from 4 pm to 5 pm and from 4 am to 5 am. This dataset contains 300 pairs of NTIR-NCV images and 300 DCV images of the same area. Using HDC-GAN, a translation experiment was performed on this UAV dataset, and Figure 14 shows the results of the translation of two images. Figure 14a,i and Figure 14c,k are the NCV and NTIR images taken by the DJI ZENMUSE XT2 camera, respectively, and Figure 14b,j is the ENCV image obtained by low-light image enhancement of Figure 14a,i. Since there are no nighttime light effects that need to be eliminated in the NCV images collected from the drone’s orthographic perspective, we use the basic MDLatLRR method to fuse ENCV and NTIR to obtain the NF image Figure 14d,l. To compare the performance of different translation algorithms, we used CycleGAN, UNSB, and HDC-GAN to translate NF images into DCV images, of which the results are shown in Figure 14e–g and Figure 14m–o, respectively. To compare the translation results, we also provide a real DCV image of the same scene taken in the afternoon (not registered with NTIR) as shown in Figure 14h,p.

As can be seen from the translation results in Figure 14, compared with CycleGAN and UNSB, the images translated by HDC-GAN maintain better semantic consistency. For example, in the first set of pictures, the four trucks on the lower right have relatively complete shapes, and the truck heads are still red. In the image translated by CycleGAN, the color of some truck heads changes to white, and all truck bodies are no longer complete, and are even confused with vegetation. The UNSB-translated image has a high degree of semantic confusion between trucks and vegetation, which are difficult to distinguish. In the second set of images, the shape and color of the vegetation in the central part of HDC-GAN-translated images are almost same as the real DCV image, while in the images translated by CycleGAN and UNSB, this part is largely translated to bare ground. In addition, the translation results of HDC-GAN maintain good semantic consistency for the blue buildings in the image, while there is a large amount of noise and blur in these buildings among the images translated by CycleGAN and UNSB.

It can be seen from this experiment that the HDC-GAN algorithm can be applied to process remote sensing orthophotos obtained in low-light environments to a certain extent, and can better improve the expression of the real environment. However, due to the extremely high requirements for the consistency of image semantics in the field of remote sensing, HDC-GAN is still difficult to achieve satisfactory results in image analysis (such as ground object classification, vegetation index calculation, etc.) when used for professional remote sensing image processing. There are two problems: firstly, the ground surface information obtained in low-light environments is insufficient, and secondly, NCV information will be further lost during the fusion process with NTIR. Therefore, the current fusion-based translation algorithm still needs further improvement for processing orthophotos of remote sensing.

6.4. Problems with HDC-GAN

Compared with other UIT algorithms, HDC-GAN has effectively minimized the introduction of ambiguous semantic information. Nevertheless, the NTIR-DCV experiment revealed the challenges associated with incorporating color information into gray-scale source images while preserving consistent semantic information. Since these two goals are contradictory, the current limitations of HDC-GAN restrict its application to the translation of NF images. In other words, color information is essential for accurately translating NTIR images.

HDC-GAN can achieve good visual effects in the translation of street scenes or images captured by UAV oblique photography. However, upon careful comparison, you will find that it is difficult to maintain strict semantic consistency in the translation of vegetation. This is prominent in the experiment in Figure 14. In fact, this problem is common to all UIT algorithms. The main reason is that green vegetation (such as leaves and grass, etc.) has strong randomness in images. This randomness is aggravated with the randomness of sample data selection during the neural network learning process, making it difficult for neural network models to handle the semantic consistency of vegetation during the translation process.

7. Conclusions

The translation of NTIR2DCV is an ill-posed problem with multiple solutions when images are translated from the temperature to the color domain. We argue that color information must be incorporated to compensate for the lack of color information in the process of TIR translation. Since visible images are easy to obtain in city environment at night, incorporating color information into NTIR images translation is feasible.

Therefore, we proposed an NCV and NTIR image fusion algorithm called IA-MDLatLRR, which weighs the illumination to improve the fusion strategy of the MDLatLRR method. The evaluation metrics SD, EN, VIF, etc. showed that our IA-MDLatLRR provided better contrast, more information content, and a better visual effect than the other methods. On the basis of ideal fusion result, we proposed the HDC-GAN model to maintain semantic consistency to improve the TIR translation result, which can avoid semantic errors that usually occur in CycleGAN, UNSB, UIT algorithms, etc. The experiment results show better quality than the UIT algorithms.

Our method is expected to be used to fuse images or videos captured by a camera like XT2 with TIR and CV lenses, and translate fusion results into visible images or videos in the daytime, which has a good application prospect in scene analysis, target detection, and other fields.