DBSF-Net: Infrared Image Colorization Based on the Generative Adversarial Model with Dual-Branch Feature Extraction and Spatial-Frequency-Domain Discrimination

Abstract

Thermal infrared cameras can image stably in complex scenes such as night, rain, snow, and dense fog. Still, humans are more sensitive to visual colors, so there is an urgent need to convert infrared images into color images in areas such as assisted driving. This paper studies a colorization method for infrared images based on a generative adversarial model. The proposed dual-branch feature extraction network ensures the stability of the content and structure of the generated visible light image; the proposed discrimination strategy combining spatial and frequency domain hybrid constraints effectively improves the problem of undersaturated coloring and the loss of texture details in the edge area of the generated visible light image. The comparative experiment of the public infrared visible light paired data set shows that the algorithm proposed in this paper has achieved the best performance in maintaining the consistency of the content structure of the generated image, restoring the image color distribution, and restoring the image texture details.

1. Introduction

Visual cameras cannot produce clear images in low-visibility environments such as night, heavy rain, heavy snow, and heavy fog. However, infrared sensors have stable imaging capabilities in such complex scenes [1,2]. As shown in Figure 1, infrared cameras can work in harsh environments such as night and smoke and can achieve all-weather and all-weather perception and monitoring of the environment. They can be used as environmental perception devices for night driving assistance systems. In applications, long-wavelength (7.5∼13 µm) infrared cameras are more robust to the interferences produced by headlights and traffic signals and a human body radiates in the long-wavelength infrared light wavelength of 9.3 µm [3], which supports the suitability of thermal cameras for capturing humans.

For the assisted driving system, infrared imaging is unaffected by low visibility environments such as nighttime and exhibits more stable perception performance. Unfortunately, infrared images have low contrast, low resolution, and no color information. According to the mechanism of human visual cognition, the human visual system can distinguish thousands of color tones and intensities but can only distinguish about twenty grayscale levels [4]. In addition, visual psychophysics experiments have suggested that, compared with pseudo-color images and grayscale images, color images can improve the speed and accuracy of the observer’s recognition of targets [5]. By using the infrared image colorization method to impart color information to the infrared image, human drivers can more clearly identify roads, obstacles, and other vehicles at night, enabling them to judge road conditions with higher accuracy while reducing the risk of accidents during nighttime driving, thereby improving the safety and visibility of nighttime driving. Therefore, the colorization of infrared images is of great significance for nighttime assisted driving.

Image colorization is an important research topic in the field of low-level computer vision, and its methods can be divided into two categories: traditional image colorization methods and deep learning-based image colorization methods. Traditional image colorization methods are semi-automatic colorization methods that require human interaction, mainly including methods based on local color extension and reference image color migration. Color extension requires the manual annotation of coloring lines in advance, while color migration requires the selection of reference color images in advance [6,7,8,9,10,11]. The reference image of the color migration method can be directly specified by the user or obtained from the Internet or from a large dataset [12,13,14,15]. Regions with similar colors or textures often have similar structures or lines, so the colorization of the image can be guided by identifying the similarities between the target image and the reference image [16,17,18,19]. The method based on reference image color migration does not require manual annotation, but relies on one or more color reference images to represent the color information. The color migration of the target infrared image is achieved by statistically analyzing the high-order statistics in the color reference image. The higher the texture and semantic information match between the reference image and the original infrared image, the better the color transfer effect. These traditional methods are not only complex in terms of their calculation process but also have a high calculation cost, making it difficult to achieve real-time colorization.

Deep neural networks have realized the automation of image colorization, effectively reducing the time required for the colorization process, and are increasingly favored by researchers [20,21,22,23]. The colorization method based on deep learning automatically obtains the mapping relationship between data, continuously optimizes the model weights, and significantly improves the generalization ability, colorization effect, and real-time performance of the algorithm. At the same time, there are also certain problems with the automated colorization method. When performing colorization tasks based on convolutional neural networks (CNNs), the information extracted by the CNN structure is translation invariant, and the weights of the extracted image background features and detail features are equal [24,25,26,27], but the understanding of image features is insufficient, resulting in a decrease in network performance and problems such as blurred and disconnected coloring. In order to make the network focus on more prominent features during deep learning, attention structures represented by Transformer are increasingly being applied by researchers to colorization tasks. Transformer uses color upsampling for colorization [28]. However, during the training process, color errors continue to accumulate and increase, resulting in a decrease in colorization accuracy and saturation.

The colorization structure based on the generative adversarial mechanism is still the most widely used approach with the best colorization effect and the most stable network structure. The image colorization method based on a generative adversarial structure [29,30,31] may cause changes in the image content and object structure due to insufficient constraints during the game between the generator and discriminator, and it may also result in edge blurring, line distortion, and other problems in the generated image. Therefore, in order to achieve a better infrared image colorization effect, this study improves and optimizes it based on the generative adversarial framework. The key contributions of this study can be summarized as follows: Firstly, this study proposes a dual-branch feature extraction network for the infrared image colorization method. The proposed method incorporates a dual-branch feature extraction structure in generative adversarial networks, extracting both basic and detailed features separately, effectively capturing long-range dependencies while preserving the original texture details as much as possible, thereby reducing edge blur and distortion in generated images. By comparing the loss through global and local saliency self-supervision, better consistency between the generated image and the original content and structure can be maintained. Secondly, this study proposes a colorization method that combines spatial-domain and frequency-domain discrimination strategies. The proposed method utilizes spatial discriminator constraints to generate the image color distribution, solving the problems of color saturation and inaccuracy. Meanwhile, the frequency-domain discriminator extracts detailed features of different frequency bands, making the generated color images more realistic and natural. The experimental results suggest that the method outperforms the existing methods in restoring image color distribution and texture detail information.

This paper proposes a dual-branch feature extraction and spatial-frequency domain discrimination network, named DBSF-Net. By combining dual-branch feature extraction with the discrimination strategy of spatial-domain and frequency-domain information, the DBSF-Net is proposed for achieving high-quality infrared image colorization. The DBSF-Net exhibits excellent performance on different infrared visible light datasets, achieving better quantitative and qualitative comparison results than the existing state-of-the-art (SOTA) colorization algorithms. Meanwhile, by deploying algorithms on embedded computing platforms, real-time infrared image colorization can be achieved in real scenarios, providing support for the real-time application of algorithms in tasks such as autonomous piloting.

2. Related Work

Deep learning-based image coloring methods can be mainly categorized from a network perspective into CNN-based methods, generative adversarial networks (GAN)-based methods, and Transformer-based structures, etc.

2.1. CNN-Based Methods

CNN was first applied to the task of infrared image colorization. Cheng et al. first proposed a fully automatic colorization method based on CNN [24]. Although this approach is effective, it still requires a large amount of training data and its performance is prone to be limited by the training dataset. Iizuka et al. used a data-driven CNN method that combines long-range global and short-range local features to realize fully automatic colorization [25]. Larsson et al. reported a self-supervised automatic coloring method which realizes colorization by predicting pixel color histograms and extracts multi-scale features using convolutional deep networks [26]. Zhang et al. regarded colorization as a classification task to increase color diversity [27]. Zhang et al. proposed a real-time user-guided colorization architecture which is subdivided into local and global prompt networks [28]. Through adopting similarity sub-networks and coloring sub-networks, He et al. implemented different colorization styles based on reference images, calculated similarity, and refined colors [32]. Subsequently, Zhang et al. applied the above method to colorize grayscale videos, introducing temporal consistency loss to reduce color fluctuations [33]. Dabas et al. used a VGG network for fully automatic image coloring [34]. Dong et al. acquired three-dimensional weight volumes based on convolutional networks, achieved high-precision colorization, and updated distorted information maps using Markov random field methods [35]. Pang et al. proposed the SFAC algorithm, which uses structural feature alignment to colorize old photos, eliminating big data dependencies and ensuring that semantically related objects have similar colors [36].

2.2. GAN-Based Methods

The most commonly used method for image colorization is GAN. Goodfellow et al. first proposed GAN, which generates realistic color images through the adversarial training of generators and discriminators [29]. Later, Isola et al. proposed a conditional adversarial network for image transformation tasks [30]. Zhu et al. first proposed CycleGAN, which enables the self-supervised image cross-domain transformation of unpaired data [31]. Suárez et al. reported DCGAN for near-infrared image coloring [37]. By maintaining the routing distance among samples to achieve good mapping performance, Benaim et al. proposed DistanceGAN [38]. Subsequently, Bansal et al. proposed RecycleGAN for unsupervised video colorization, overcoming the challenge of infrared video colorization [39]. Kniaz et al. proposed ThermalGAN for cross-modal color thermal infrared facial re-recognition [40]. Mehri et al. proposed a near-infrared image colorization method based on unpaired datasets [41]. Abbott et al. introduced adaptive CycleGAN for the conversion of long-wave infrared and visible light datasets [42]. Emami et al. proposed SPA-GAN, which guides the generator to focus on key image regions and generate more realistic images through discriminative salient feature scores [43]. Chen et al. proposed a GAN without independent encoders, which simplifies the structure and directly trains the encoder using the adversarial loss of multi-scale discriminators to improve training efficiency [44]. Park et al. utilized an image transformation framework based on contrastive learning to increase mutual information [45]. Han et al. proposed an unsupervised image transformation method based on the contrastive learning of dual-branch encoders, which exhibits excellent performance [46]. Huang et al. combined U-Net and skip connections to improve the cyclic consensus GAN for the automatic colorization of images and used a composite loss function to enhance the authenticity of color images [47]. Li et al. proposed an unsupervised infrared video conversion method that constrains the video conversion process through perceptual period loss and region similarity loss [48]. Yadav et al. proposed an efficient attention recursive GAN based on MobileNet for the conversion of resource-limited infrared to visible light [49]. Luo et al. designed PearlGAN, which includes an attention-guided module and a structured gradient alignment loss to promote edge consistency [50]. Yu et al. proposed the ROMA framework to convert unpaired nighttime infrared videos into fine-grained daytime visible light videos and designed a multi-scale region discriminator [51]. Guo et al. proposed structural consistency constraints to improve image structural consistency [52]. Lin et al. designed an image structure preservation method based on cyclic consistency to alleviate the problem of insufficient supervised labels in source-domain images [53]. Bharti et al. proposed a multi-objective recurrent adversarial network with quantized evolutionary gradient perception, which uses evolutionary computation and multi-objective optimization methods to improve coloring efficiency [54].

2.3. Transformer-Based Structures

To learn key image features, researchers have incorporated attention mechanisms into neural networks to improve network performance. Transformer-based attention mechanisms have been widely adopted for image colorization tasks. Zhao et al. proposed a new cyclic switching GAN combining Swin Transformer and convolutional layers to improve the colorization effect of infrared images [55]. Feng et al. proposed a multi-scale training structure and a progressive growth generator method to generate fine-grained style images through cross-attention mechanisms [56]. Guo et al. proposed multi-feature contrastive learning to improve discriminator performance and solve model collapse problems [57]. Liu et al. proposed the TCVC network to enhance the temporal consistency of video coloring using a self-normalization learning scheme that does not require training with ground truth color videos [58]. Liang et al. introduced color control and multi-mode colorization methods, utilizing pre-trained stable diffusion models to achieve accurate local color operations [59]. Wei et al. proposed a new infrared colorization algorithm that achieves cross-modal zero-shot learning through frequency-domain feature decoupling and reconstruction without requiring infrared datasets for training [60]. Kumar et al. proposed the ColTran grayscale coloring network architecture based on Transformer blocks, which provides a variety of color grayscale images through the construction of autoregressive colorimeters, color up-samplers, and spatial up-samplers [61]. Kim et al. proposed the InstaFormer model, which integrates global and instance-level information [62]. The ColorFormer proposed by Ji et al. automatically colors through a mixed attention mechanism assisted by color memory [63]. Zheng et al. proposed an efficient image transformation structure which includes a hybrid perception block and a dual-pruning self-attention module [64]. Torbunov et al. utilized Visual Transformer (ViT) to achieve a high correlation between the original image and the translated image [65]. Ma et al. proposed CFFT-GAN, which improves image conversion performance by decoupling and fusing features through cascaded CFFT modules [66]. Jiang et al. proposed the MATEBIT structure to learn cross-domain correspondence relationships and enhance feature acquisition for high-quality images [67]. Lee et al. proposed an efficient ViT architecture for real-time interactive coloring which realizes acceleration by pruning redundant image blocks and layers [28]. Chen et al. proposed a sample-based video colorization framework with long-term spatiotemporal dependencies which generates more diverse, realistic, and stable results [68].

3. Materials and Methods

3.1. Overall Framework of the Proposed Algorithm

The study proposes a colorization method for infrared images based on a dual-branch feature extraction network. Specifically, the dual-branch feature extraction structure is integrated into the GAN to extract both basic and detailed features, and a global and local saliency self-supervised contrastive loss is established by adding a saliency feature query modules to ensure the invariance of the generated image content and structure. This method improves the existing GAN-based colorization method, which has the problem of adding or deleting the original image content and changing the original object structure in the generated color image, and maintains the invariance of the generated content and structure. In addition, due to the limitations of the imaging mechanism of infrared cameras, the grayscale values of neighboring pixels have similarities. Using pixel-level loss discrimination during colorization tasks can result in problems such as undersaturation or inaccurate coloring in the target edge region. Moreover, multiple convolutions and sampling processes in deep networks can cause the loss of the original infrared image details and texture information in the generated image. In response to the above problems, this study proposes a dual-discrimination strategy that combines the spatial domain and frequency domain to achieve higher-quality colorization. The use of spatial discriminators primarily aims at making more precise judgments on the color distribution of the generated results and real samples to solve the problems of undersaturation and inaccurate coloring. The frequency-domain discriminator mainly addresses the problem of generating color images that lose the texture details of the original infrared images. The original infrared images are subjected to multi-scale geometric analysis and transformation to obtain frequency-domain images. By effectively extracting detailed features of different frequency bands, the texture details of the generated images are enhanced.

The schematic diagram of the overall structure of the colorization algorithm network proposed in this subsection is shown in Figure 2. The proposed network is based on a generative adversarial structure in which the generator introduces a long-short distance feature extraction module based on the Transformer and CNN, according to the U-Net encoding and decoding structure, and introduces a saliency feature query module to construct global and local self-supervised contrastive losses, thereby effectively constraining the content and structure of the generated images by the generator. Firstly, the original source-domain infrared image $I_x$ is input into the U-Net encoding and decoding structure to generate a three-channel color image $G\left(I_x\right)$ through continuous downsampling and deconvolution upsampling while calculating the adversarial loss Lg during this process. The basic module of the U-Net is chosen to be Darknet to maintain the efficiency of image generation, and single-channel $I_x$ is duplicated and expanded to 3 channels before it enters the network. By encoding $I_x$ and $G\left(I_x\right)$ separately, the features $F_x$ and $F_y$ of the source-domain X and target-domain Y are obtained. Then, the source-domain and target-domain features are input into the Transformer feature extraction network, and the multi-head attention mechanism in the Transformer is used to extract the long-distance low-frequency global features of images. Meanwhile, the source-domain and target-domain features are input into the CNN feature extraction network, and the local features are extracted, preserving the texture detail information of the original source-domain image as much as possible. Subsequently, the global saliency feature query module calculates its global self-supervised comparison loss with the input, while the local saliency feature query module calculates its local self-supervised comparison loss with the input. The saliency feature query module selects the anchor point of the saliency feature position from the target-domain features and then samples a corresponding positive and negative feature from the source-domain features to calculate the comparison loss of the anchor point, through which the model can use the mutual information among the corresponding features as much as possible. Finally, a dual-discrimination hybrid strategy using spatial-domain and frequency-domain information is used to discriminate the generated color images. The discrimination process is guided by a composite loss constraint of adversarial loss and saliency contrast loss until a consistent colorization effect of the generated content structure is achieved.

3.2. The Structure of Multi-Branch Feature Extraction Network

3.2.1. Lightweight-Transformer-Based Global Feature Extractor

For the features $F_x$ and $F_y$ encoded from the input original infrared image and the generated color image, a Transformer extraction network with spatial self-attention mechanism is used to extract their global features $F_x^g$ and $F_y^g$, respectively. Specifically, the feature map can be segmented into fixed-size feature blocks, and the linear projections of these feature blocks and their corresponding image positions can be input into the Transformer encoder. In the encoder, a multi-head attention mechanism is used to calculate the attention score between each feature block and other feature blocks, thereby capturing global features. However, conventional Transformer modules require the extraction of global and local contexts, with a large model parameter scale and high computational complexity, which makes it difficult to deploy on mobile platforms. Therefore, the feature extraction network introduces the Lite Transformer (LT) [69] block as the basic unit of the global feature extraction network. Compared with the traditional Transformer structure, the overall computational complexity of the LT structure is reduced by half. While ensuring the same performance, the model is lighter and more in line with the computing power requirements of resource-constrained devices, such as vehicular platforms.

3.2.2. Inverse Neural Network (INN)-Based Detail Feature Extractor

This module introduces a local feature extractor based on the CNN architecture which extracts the local detail features $F_x^l$ and $F_y^l$ from the features $F_x$ and $F_y$, respectively, to better preserve the texture detail information of the original image and perform feature extraction as losslessly as possible. The more lightweight INN block is used as the basic unit of the detail feature extractor. INN is a coupled layer in which the activation results of each layer can be reversely inferred from the results of the next layer. Therefore, in the process of the reverse inference of the results, the INN network only saves the results of the last layer of the network and the parameters of each intermediate layer, and the results of each intermediate layer can be derived through the backpropagation algorithm. In this process, there is no need to save the activation results of each layer directly. This method can significantly reduce memory usage.

The forward propagation process of INN is shown in Figure 3a. Each layer’s input is divided into two parts, $u_1$ and $u_2$, and then transformed and alternately coupled by learning the functions F and P to obtain the output ($v_1$, $v_2$). The calculation process is shown in Equation (1).

$$\begin{array}{cc}\hfill v_1& =u_1+F\left(u_2\right)\hfill \\ \hfill v_2& =u_2+P\left(v_1\right)\hfill \end{array}$$

(1)

The backpropagation process of INN is shown in Figure 3b, in which ($u_1$, $u_2$) can be directly recovered by reverse calculation. The reverse calculation and forward calculation use the same parameters, and the reverse calculation process is described by

$$\begin{array}{cc}\hfill u_2& =v_2-P\left(v_1\right)\hfill \\ \hfill u_1& =v_1-F\left(u_2\right)\hfill \end{array}$$

(2)

where F and P are similar residual functions without requiring reversibility.

Therefore, the INN blocks must be successively connected, amongst which any other network modules must not be inserted to prevent information loss and reversible computing processes from being interrupted.

3.2.3. Saliency Query Module

The saliency query module is introduced to calculate the contrast loss. The learning process is constrained by global and local contrast losses to solve the problem that the original content and structure are changed during image colorization.

(1) Global self-supervised saliency query module

The global self-supervised saliency query module combines contrastive learning between the source-domain global feature $F_x^g$ and the target-domain global feature $F_y^g$ extracted by the feature extraction network. The overall structure of the global query module is shown in Figure 4. Firstly, the feature map $F_x^g\in {\mathbb{R}}^{h\times w\times c}$ (h, w, c represent the height, width, and channel count of the feature, respectively) is reshaped into the query matrix $Q\in {\mathbb{R}}^{hw\times c}$, while the key matrix $K\in {\mathbb{R}}^{c\times hw}$ is obtained. Then, conducting Softmax activation with respect to the query Q will output the global saliency attention matrix $B_g\in {\mathbb{R}}^{hw\times hw}$. After that, the saliency of the corresponding features is measured by calculating the entropy $H_g$ of each row in $B_g$, which is described as

$$H_g\left(m\right)=-\sum _{n=1}^{hw}{B}_g(m,n)log{B}_g(m,n),$$

(3)

where $B_g\in [0,1]$, which is the output of the softmax layer. m and n are the indices of Q and K, respectively, corresponding to the rows and columns in the global saliency matrix.

Entropy is used to describe the degree of disorder in a vector. When one element in $H_g\left(m\right)$ is close to 1 and the rest are close to 0, the entropy approaches 0. Thus, a lower $H_g\left(m\right)$ indicates a stronger saliency of the feature. For our proposed algorithm in this study, it is necessary to select all salient queries. Sort the rows of the global saliency attention matrix Bg in ascending order of their entropy $H_g$, and select the N rows with the lowest entropy to construct a new matrix $B_g^N\in {\mathbb{R}}^{n\times hw}$.

Furthermore, based on the saliency attention matrix $B_g^N$, the saliency feature positions are selected from $F_y^g$ as anchor points q, and $F_x^g$ is routed to obtain a positive feature value $v^{+}$ and (N-1) negative feature values $v^{-}$. Based on the obtained routing values and temperature hyper-parameter $\tau$, a global self-supervised contrastive loss $L_c^g$ is constructed to ensure the consistency of source-domain feature relationships in the generated results. The calculation of $L_c^g$ is expressed as

$$L_c^g=-log\left[1-{\displaystyle \frac{{\sum }_{i=1}^{N-1}exp(q_i·{\mathrm{v}}_i^{-}/\tau )}{exp(q_i·{\mathrm{v}}_i^{+}/\tau )+{\sum }_{i=1}^{N-1}exp(q_i·{\mathrm{v}}_i^{-}/\tau )}}\right]$$

(4)

(2) Local self-supervised saliency query module

Similarly, the local self-supervised saliency query module combines contrastive learning between the source-domain local feature $F_x^l$ and the target-domain local feature $F_y^l$ extracted by the feature extraction network. The local query module mainly measures the similarity between queries and adjacent keys within a fixed-size (s × s) local area, which is used to capture spatial features of the local area. Using the same global attention module, reconstruct $F_x^l$ as $Q_l\in {\mathbb{R}}^{hw\times c}$ with a key matrix of $K_l\in {\mathbb{R}}^{hw\times s^2\times c}$. Multiply these two matrices and perform softmax regression to obtain the local attention matrix $B_l\in {\mathbb{R}}^{hw\times s^2}$, and calculate the local entropy of each row according to the following Equtation (5).

$$H_l\left(m\right)=-\sum _{n=1}^{s^2}{B}_l(m,n)log{B}_l(m,n)$$

(5)

Sort $H_l$ in ascending order, select the N rows with the smallest $H_l$ as the matrix $B_l^N$, and further route to obtain $v_l^{+}$, $v_l^{-}$ and $q_l$. Construct the local saliency comparison loss $L_c^l$:

$$L_c^l=-log\left[1-{\displaystyle \frac{{\sum }_{i=1}^{N-1}exp(q_{li}·v_{li}^{-}/\tau )}{exp(q_{li}·v_{li}^{+}/\tau )+{\sum }_{i=1}^{N-1}exp(q_{li}·v_{li}^{-}/\tau )}}\right]$$

(6)

3.2.4. Discriminator Based on Spatial-Domain and Frequency-Domain Information

As shown in Figure 5, the source-domain infrared image $I_x$ and the target-domain real-color image $I_y$ are simultaneously input into a colorization generator to produce the colorized image $G\left(I_x\right)$. Next, $G\left(I_x\right)$ and the real-color image $I_y$ are fed into a Markov (PatchGAN) spatial discriminator, which ensures that the generated image shares the same background and color distribution as the real image. The PatchGAN discriminator effectively addresses issues like inaccurate coloring and undersaturation and can effectively capture local spatial features. Furthermore, a frequency-domain discriminator based on Haar is used to extract frequency-domain features. $G\left(I_x\right)$ and the original infrared image $I_x$ are input into a frequency-domain discriminator, based on Haar wavelet transforms, to ensure the generated image preserves the texture details of the infrared image, thereby solving problems such as line distortion and blurred details.

(1) Spatial-domain discriminator

The structure of the spatial discriminator D1 is shown in Figure 6, which introduces learning process constraints based on Markov discriminators. Specifically, this discriminator uses six stride convolutions with a kernel size of 5 and a stride size of 2. These convolutional layers can effectively capture the statistical feature information of Markov image patches. Firstly, the PatchGAN maps the input image to N × N patch matrices and then characterizes the local features of the image by using patch block discrimination. Finally, the final classification results are averaged to obtain an overall discrimination result. In this way, the discriminator can finely distinguish the local features of the image, thereby guiding the generator to generate more accurate and uniform-color images.

(2) Frequency-domain discriminator

The frequency-domain discrimination strategy based on the Haar wavelet transform is adopted to retain sufficient texture and detail information of the original infrared image $I_x$. Firstly, the original infrared image $I_x$ and the generated image $G\left(I_x\right)$ are decomposed into a base sub-image and three high-frequency sub-images using the Haar transform. These sub-images are then concatenated together and sent to the frequency-domain discriminator $D_2$ for discrimination. The structure of $D_2$ is shown in Figure 7. Grouped convolution is used to extract features from the four sub-band images in $D_2$. Considering the different importance of each frequency component in the feature representation of infrared images, Squeeze-and-Excitation (SE) is used to weigh the features of different frequency bands. Therefore, in the identification process of $D_2$, the texture details in the generated image will be enhanced.

(a) The Haar wavelet transform

The equation of the wavelet transform can be expressed as

$$WT(a,\tau )={\displaystyle \frac{1}{\sqrt{a}}}\underset{-\infty }{\overset{+\infty }{\int }}f\left(t\right)\ast \psi \left({\displaystyle \frac{t-\tau }{a}}\right)dt$$

(7)

where a denotes the scaling scale, $\tau$ denotes the translation of the primary function, t denotes the time, $f(·)$ denotes the time-domain signal, and $\psi (·)$ denotes the wavelet’s primary function. The mother wavelet of the Haar wavelet transform is expressed by

$$\psi \left(t\right)=\left\{\begin{array}{cc}& 1,0\le t<1/2\\ & -1,1/2\le t<1\\ & 0,\mathrm{else}\end{array}\right.$$

(8)

the corresponding father wavelet to which is expressed by

$$\phi \left(t\right)=\left\{\begin{array}{c}10\le t<1\\ 0,\mathrm{else}\end{array}\right.$$

(9)

In order to meet the requirement of orthogonality between the father wavelet and the mother wavelet, a filter is required, which can be described as

$$\mathrm{h}\left[n\right]=\left\{\begin{array}{c}1/\sqrt{2}n=0,1\\ 0,\mathrm{else}\end{array}\right.$$

(10)

Based upon the above Equation (10), we obtain

$$\psi \left(x\right)=\phi \left(x\right)-\phi (2x-1)$$

(11)

The Haar primary function obtains signal scale information, while the Haar wavelet function represents signal detail information. The Haar wavelet has symmetry and only takes the values of 1 and −1, which are easy to calculate.

(b) Squeeze-and-Excitation module

The Squeeze-and-Excitation (SE) block [70] is an information feature construction module used in CNNs which calibrates channel feature maps by constructing dependency relationships between channels. The overall structure of the SE block is shown in Figure 8. Firstly, the input information X is mapped to the feature U through a specific F-transform. The F-transform can be a convolution operation, pooling operation, or other linear transformation operations, depending on the requirements and design of the network. Then, feature U passes through a squeezing layer, which performs channel-level feature aggregation mapping in the spatial dimension (H × W), compressing each channel in feature U into a one-dimensional vector, which is called a channel descriptor. The squeezing operation can effectively reduce the number of channels in feature maps, thereby minimizing computational complexity and model complexity. Then, the channel descriptor passes through the excitation layer, which adopts a self-selected channel mechanism to directly combine high-dimensional features as input and generate a set of channel weights. The high-dimensional feature combination is a fixed-length vector, which can be randomly initialized or pre-trained. The function of the incentive layer is to generate channel weights for each dimension based on the combination of high-dimensional features and channel descriptors and to perform weighted summation on the feature map U to obtain the output tensor of the SE. The output tensor of the SE block can be directly input into other networks or combined with other building blocks to construct more sophisticated network structures.

3.3. Loss Function

In order to effectively constrain the learning process of the colorization network, a composite loss function is designed to guide training, so that the generated colorized image maintains consistency with the original infrared image in terms of its content structure. The specific composite loss includes the adversarial loss of the generator, the global self-supervised contrastive loss, the local saliency contrastive loss, and the discriminator mean square error (MSE). The overall composite loss function is expressed as

$$L_{Total}=L_g+{\lambda }_1{L}_c^g+{\lambda }_2{L}_c^l+{\lambda }_3{L}_{MSE}$$

(12)

where ${\lambda }_1$, ${\lambda }_2$ and ${\lambda }_3$ represent the hyper-parameter weights of the global saliency contrastive loss, local saliency contrastive loss, and the MSE loss function, respectively.

The adversarial loss generated by the generator can ensure the overall coloring effect of the colorization network and obtain a color image with true overall color. The contrast loss of global and local saliency is used to compare the global basis and local details, respectively, ensuring the consistency of the content and structure through the contrast loss while colorizing the background and detail information of the results. The generated adversarial loss is expressed as Equation (13):

$$L_g=E_{I_x\in X}log(1-D\left(G\left(I_x\right)\right))+E_{I_y\in Y}logD\left(I_y\right)$$

(13)

where $D(·)$ denotes the discriminator operation, E stands for the expectation operator, $G(·)$ denotes the generator operation, $I_x$ denotes the raw input infrared data, and $I_y$ denotes the raw input visible light data. The global self-supervised contrastive loss $L_c^g$ and the local saliency contrastive loss composite $L_c^l$ are described in Section 3.2.3. Finally, in order to constrain the generator to generate samples approximating the real samples, the MSE is specifically expressed in Equation (14):

$$L_{MSE}=-{\displaystyle \frac{1}{N}}{{\sum }_{i=1}^N(G\left(I_{x_i}\right)-I_{y_i})}^2$$

(14)

where N denotes the number of samples, G(Ix) denotes the generated image, and Iy denotes the real-color image.

4. Experiment and Results

Comparative experiments were conducted on the proposed colorization algorithm and the selected comparison methods. Firstly, the datasets used in these experiments and the details of the experimental implementation are introduced. Then, the proposed method is qualitatively and quantitatively compared with other comparative methods in different experiments. In addition, ablation experiments are conducted to demonstrate the effectiveness of important components such as the saliency query module and the spatial-frequency domain information discriminator. Finally, considering the limited computing resources of the autopilot platform, the efficiency of different comparison methods is evaluated on the commonly used edge computing platform of NVIDIA DRIVE AGX Orin (ARM+GPU architecture) in assisted driving applications.

4.1. Implementation Details

Experimental environment:

This subsection conducts comparative experiments on the colorization algorithm and selected comparison methods studied and implemented in this section. The training and testing of the experiments are based on the supercomputer server equipment in the laboratory, which integrates a total of three 2080Ti 11GB graphics cards, (The manufacturer of the 2080Ti 11GB graphics cards is NVIDIA, with the company headquarters located in Santa Clara, CA, USA.) including a system disk and two storage disks. The operating system is Ubuntu 16.0.4, and the algorithm running environment requirements in this study includes deep learning libraries such as Python 3, Torch 1.6, Torchvision 0.7.0, Dominate 2.4, etc.

Experiment data: The datasets used for colorization network training include publicly available datasets (KAIST) [71]. The KAIST is a pedestrian dataset in which the infrared images are captured by FLIR-A35 at wavelengths of 7.5–13 µm. The dataset consists of 95,328 sets of images, each containing a pair of color and thermal infrared images, which use beam splitter-based hardware to physically align the two image domains, thus covering the same spatial locations. The intrinsic spatial resolutions of the visible RGB image and the infrared image are the same. This dataset provides various conventional traffic scenarios such as campuses, streets, pedestrians, etc., during day and night. The image size in the dataset is 640 × 480, and the dataset includes 12 sets of folder data. Figure 9 shows partially paired images from the KAIST dataset.

Experimental setup: Two different datasets are used, one from the KAIST and the other from the self-built dataset. In total, 4000 pairs of infrared and visible light data are randomly selected as the training set for each dataset, and 1000 pairs are randomly selected for constructing the test set. All image sizes are set to 256 × 256 to ensure input consistency. During the training stage of the proposed algorithm, set the traversal epochs to 200 and the batch size to 1. In order to effectively learn the model parameters, this study sets the initial learning rate to 0.0002 and saves the weight model every 5 epochs for the purpose of saving and backtracking the model state during the training process. The entire training and testing process of the model is based on the PyTorch learning framework, which uses a single computer and a single graphics card (1 GPU) for computation to ensure the efficiency and accuracy of the training process. In Equation (12), the hyperparameters ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ are set to 0.5, 0.5, and 1, respectively. These were determined through experimentation to be the optimal weights.

Evaluation criterion: The quantitative indicators used in this study include peak signal-to-noise ratio (PSNR), which evaluates the quality of the generated samples, and the structural similarity (SSIM), which evaluates the similarity between the generated samples and real samples, as well as the learning perceptual image patch similarity (LPIPS) and vision information fidelity (VIF), which are evaluation indicators based on human vision, thereby evaluating the quality of the output images of the generated model.

(a) PSNR: The PSNR is an indicator for measuring the quality of the generated images, and the higher the PSNR value, the better the quality of the generated image. The specific process of calculating the PSNR first requires obtaining the MSE of the image (as shown in Equation (15)) and performing logarithmic operations based on MSE to obtain the PSNR index. The PSNR unit is dB. Given the generated image Y and the reference sample X, the larger the PSNR calculation result, the better the algorithm’s coloring effect.

$$\mathrm{MSE}={\displaystyle \frac{1}{HW}}\sum _{i=0}^{H-1}\sum _{j=0}^{W-1}{∥I_{\mathrm{y}}(i,j)-I_x(i,j)∥}_2^2,$$

(15)

The PSNR calculation is expressed by

$$\mathrm{PSNR}=10\times {log}_{10}\left({\displaystyle \frac{2^c-1}{\mathrm{MSE}}}\right),$$

(16)

where c denotes the number of image bits, H denotes the height of the image, and W denotes the width of the image.

(b) SSIM: The SSIM is an indicator used to evaluate the structural similarity between the generated samples and real samples. The SSIM mainly calculates similarity differences based on three parameters: the pixel values, contrast, and overall image structure between two compared images. The SSIM value is between [0, 1]; the larger the SSIM value, the more similar the generated sample is to the reference sample. The SSIM calculation can be expressed as Equation (17):

$$\mathrm{SSIM}(I_x,I_y)={\displaystyle \frac{(2{\mu }_{I_x}{\mu }_{I_y}+c_1)(2{\sigma }_{I_x{I}_y}+c_2)}{({\mu }_{I_x}^2+{\mu }_{I_y}^2+c_1)({\sigma }_{I_x}^2+{\sigma }_{I_y}^2+c_2)}},$$

(17)

where $\mu$ denotes the sample mean, $\sigma$ denotes the sample standard deviation, ${\sigma }_{xy}$ denotes the sample covariance of Ix and Iy, and c1 and c2 are two constants.

(c) LPIPS [72]: Also known as “perception loss,” the LPIPS is more in line with human visual perception than the commonly used PSNR and SSIM indicators for image evaluation. The lower the LPIPS value, the more similar the two sample data are. The LPIPS can be calculated by

$$D\left(I_y,I_x\right)=\sum _l{\displaystyle \frac{1}{H_l{W}_l}}\sum _{h,w}\left|\right|w_l\odot \left({\widehat{y}}_{hw}^l-x_{hw}^l\right){\left|\right|}_2^2,$$

(18)

where D denotes the distance between the generated sample and the reference sample. Separate the feature stack from the l-th layer and perform unit normalization. Meanwhile, adjust the number of activated channels in use and calculate the L2 distance. Then, calculate the average in space and the sum in the channels.

(d) VIF [73]: The VIF is an indicator used to evaluate the visual quality and details of the original image in digital image processing which is based on natural scene statistical estimation and human visual distortion modeling. The value of the VIF index is obtained according to the calculation of mutual information, and the value of the VIF index is between [0, 1]. The larger the index value, the better the image quality.

4.2. Experimental Results

According to the experimental data, the colorization conversion effect of the proposed algorithm and the comparison method is verified and tested. The comparative experiments mainly select CycleGAN [31], MocycleGAN [74], ToDayGAN [75], CUT [45], and DECENT [46]. Five coloring algorithms are compared and validated under the same experimental conditions, trained on the same dataset, and tested for coloring effects in different scenarios.

For the selected KAIST experimental dataset, this subsection randomly selects data from various scenarios for comparative experimental training and testing. The colorization effect of various algorithms on infrared images is evaluated through evaluation indicators. The experimental comparison results of this section are shown in Figure 10 and Figure 11, covering two scenarios: road and traffic conditions. Meanwhile, subjective and objective evaluations are conducted on both the comparative experiment and the experimental results, the evaluation results of which are listed in

Table 1. KAIST “Traffic” scenario indicator quantitative results.

Indicator	CycleGAN	ToDayGAN	MocycleGAN	CUT	DECENT	Ours
PSNR ↑	13.3907	10.6866	13.9347	14.2114	15.2230	15.7025
SSIM ↑	0.3091	0.2952	0.4075	0.3659	0.3035	0.4038
LPIPS ↓	0.4109	0.5542	0.4647	0.3528	0.3831	0.3637
VIF ↑	0.7827	0.8057	0.8040	0.8026	0.8068	0.8162

The bolded data indicate the best results.

,

Table 2. Indicator result of “Road1” scene in KAIST.

Indicator	CycleGAN	ToDayGAN	MocycleGAN	CUT	DECENT	Ours
PSNR ↑	14.1729	10.3912	10.0528	12.3128	17.5589	17.6733
SSIM ↑	0.4584	0.2694	0.2245	0.3153	0.5026	0.5697
LPIPS ↓	0.4113	0.5733	0.5986	0.4287	0.4952	0.3868
VIF ↑	0.7907	0.7788	0.7934	0.7864	0.8063	0.8471

and

Table 3. Indicator result of “Road2” scene in KAIST.

Indicator	CycleGAN	ToDayGAN	MocycleGAN	CUT	DECENT	Ours
PSNR ↑	13.6457	10.5213	11.5467	12.5498	16.8880	17.4392
SSIM ↑	0.4408	0.2441	0.3059	0.3102	0.5798	0.6123
LPIPS ↓	0.4329	0.6035	0.5130	0.4422	0.2965	0.2919
VIF ↑	0.8266	0.8039	0.8026	0.8073	0.8400	0.8475

. It can be observed from these tables that the bold values represent the optimal results. These experimental results are of great significance for evaluating the performance of various algorithms in colorization conversion and help to better understand and compare the effects and characteristics of each algorithm.

(1) Experimental results and analysis based on KAIST “Traffic” scenario

Qualitative Evaluation: The colorization results for the “traffic” scenario in the KAIST dataset are depicted in Figure 10. From the perspective of overall image color, each method exhibits a similar color distribution to the original visible light image, with the exception of the DECENT method, which does not show any significant coloration errors. Then, upon observing the road marking lines, in the colorization result of the DECENT method, the position of the double yellow lines on the ground is noticeably incorrect, with lines distorted and disconnected from the distant double yellow lines. Additionally, the colorization position of the white dividing lines on the ground is noticeably incorrect. Near the position of the double yellow lines, the MocycleGAN and CUT methods also produce some extraneous content. Observing the ground shadow areas, the colorization results of CycleGAN, ToDayGAN, MocycleGAN, CUT, and other methods have issues with the absence of color and shape imbalance in the ground shadow areas. In the DECENT method’s colorization result, the color of the ground shadow area overflows into the original area, and the shape distribution is completely unbalanced. However, the colorization shape of the ground shadow area proposed in this section is relatively consistent with the original image, with clear colors and less noise. From the perspective of the building area, the reconstruction results of this method have clear edges and richer texture details, capable of reconstructing the overall scene of pedestrians and the supermarket entrance in the roadside area on the first floor. The building area of the roadside area on the first floor reconstructed by CycleGAN, ToDayGAN, MocycleGAN, and other methods is blurred, making it difficult to distinguish people and scenes. The scene reconstructed by the DECENT method in this area is very blurred, with colors overlapping with other areas, making it impossible to distinguish the scene. Observing the roadside cars, the reconstruction results of the CycleGAN, ToDayGAN, MocycleGAN, and DECENT methods show distorted car shape content and blurred car images, making them difficult to recognize. The shape and color of the cars reconstructed by this method are roughly consistent with the original visible light image. Finally, observing a row of roadside trees, the tree area in the reconstruction results of MocycleGAN and DECENT is color-bonded together, losing the original shape and details of the trees. In summary, the method proposed in this section maintains consistency with the original image in content and structure, retains more texture and detail information, and has not experienced large-scale color overflow or shadow errors.

Quantitative evaluation: By analyzing the comparative results of various metrics in Table 1, it can be observed that the proposed method has optimized the PSNR metric by approximately 3% compared with the next best result, with the generated images having the smallest error compared with the reference images. In terms of the SSIM index, the proposed method has optimized the next best result by about 6%, with the overall structure of the reconstruction results being closer to the original visible light images. Regarding the VIF index, the proposed method has improved by about 1% compared to the next best result. The proposed method outperforms other algorithms in terms of the PSNR, SSIM, and VIF metrics. In the LPIPS index, the difference between the proposed method and the optimal method is very small. In summary, the proposed method performs satisfactorily in both subjective and objective assessments.

(2) Experimental results and analysis based on KAIST “Road1” and “Road2” scenarios

Qualitative evaluation: As shown in Figure 11, from the colorization results of the “Road1” scene images in the KAIST dataset, each method maintains a color distribution that is roughly consistent with that of the original visible light images. However, upon observing the figures in the image, the reconstruction results of the CycleGAN, ToDayGAN, MocycleGAN, and DECENT methods exhibit inconsistencies in the content and structure of the figures compared with the original images. For instance, the figures reconstructed by MocycleGAN lost the lower leg content, while the CUT method and the proposed method essentially reconstructed the general shape of the figures, with the edges of the figures in the reconstruction results of the proposed method being clearer and more distinct. Observing the roadside steps, the distant steps in the reconstruction results of CycleGAN and ToDayGAN did not have the corresponding colors reconstructed. The steps in the MocycleGAN reconstruction results were blurred, losing edges and details. In the CUT method’s reconstruction results, line distortion occurred in the distant steps. In the reconstruction results of the proposed method, the steps were almost completely reconstructed, with the color distribution being essentially consistent with the original image. Observing the streetlight area in the image, the reconstruction results of the CycleGAN, MocycleGAN, CUT, and DECENT methods indicate that the distant streetlights are not prominent, with content loss, making it difficult to fully distinguish their shapes. However, in the reconstruction results of the proposed method, the distant streetlights are relatively more prominent. From the perspective of observing the ground marking lines, compared with other baseline algorithms, the reconstruction results of the proposed method show clearer edges of the marking lines, with relatively balanced line shapes, and the highest similarity to the original image. Observing the red shrub area on the left side of the image, the reconstruction results of CycleGAN, ToDayGAN, and MocycleGAN failed to reconstruct this part of the content. The colorization position of the infrared shrubs in the DECENT reconstruction results is incorrect, while both the CUT method and the proposed method can correctly reconstruct the position of the red shrubs.

Quantitative evaluation: In the analysis of the comparative results of various indicators in Table 2 and Table 3, for the “Road1” and “Road2” scenarios, the proposed method has achieved the best performance, generating images with the highest quality and lowest distortion, and the overall structure has higher similarity. Furthermore, judging from the aforementioned metrics, our generated images have colors that align more closely with human basic cognition compared with those from methods based on color lookup tables, and they exhibit superior textural characteristics.

(3) The efficiency of different comparison methods

Considering the assisted driving vehicles’ load capacity and power supply, the performance of computing platforms is limited. Thus, all of the comparison methods’ efficiency must be taken into consideration. The efficiency of all the methods is compared on the NVIDIA DRIVE AGX Orin, and the results are shown in

Table 4. Efficiency of different comparison methods.

Indicator	CycleGAN	ToDayGAN	MocycleGAN	CUT	DECENT	Ours
Parameter(M) ↓	28.286	83.171	42.147	14.703	57.183	12.371
Runtime(s) ↓	0.251	0.783	0.374	0.137	0.243	0.081

.

The methods with fewer parameters (M means megabytes of memory) that still manage shorter runtimes are considered highly efficient. The experimental results show that the proposed method in this paper has a minimal amount of parameters and computation and higher inference speed compared with the different comparison methods. In physical experiments, only the image network generation module participates in the computation. This module is based on the Darknet architecture to ensure efficient image generation. With the acceleration of the TensorRT engine, it can achieve an inference speed of over 15 frames per second on the in-vehicle intelligent inference terminal NVIDIA DRIVE AGX Orin. The results mean the proposed is suitable for applications with strict power and speed constraints like assisted driving.

4.3. Ablation Experiment

To further validate the effectiveness of each module of the improved method, ablation experiments are conducted on the KAIST dataset. The baseline network used is the QS-Attn network, which is based on a generative adversarial structure and also uses U-Net as the generator, with spatial-domain information (PatchGAN) forming the discriminator. Based on the baseline network, this study introduces a local feature extraction network based on CNN and establishes a corresponding local saliency contrast loss. Meanwhile, a global feature extraction network is introduced based on Transformer and a global saliency contrastive loss is thus established. To verify the effectiveness of local and global feature extraction networks, the ablation experiment is presented as follows: (1) Baseline QS-Attn model without local and global feature extraction network modules; (2) baseline model with local feature extraction module and local saliency comparison loss; (3) baseline model with global feature extraction module and global saliency comparison loss; (4) the proposed method. The results of the ablation experiment are shown in Figure 12. The objective evaluation indicators are listed out in

Table 5. Objective indicators for infrared image colorization using different modules.

Methods Compared	PSNR ↑	SSIM ↑	LPIPS ↓	VIF ↑
Baseline	17.1539	0.5285	0.3457	0.8320
Baseline + local feature extraction	18.7583	0.5580	0.3123	0.7947
Baseline + global feature extraction	20.1790	0.6102	0.2921	0.8424
Baseline + baseline + global + local	20.2499	0.6114	0.2769	0.8438
Ours	20.5134	0.6156	0.2407	0.8404

, in which the bolded data indicate the optimal results.

Qualitative evaluation: The ablation comparison experiment results for the improved modules in the “Road” scenario of the KAIST dataset are shown in Figure 12. Compared with the baseline method, the addition of a local and global feature extraction module improved the restoration effect of the road surface font in the road surface representation part of the image. However, compared with the baseline method, the proposed method optimized the generated image road surface font to be relatively more complete. Additionally, observing the zebra crossing area in the image, both the baseline method and the reconstruction results with local feature extraction have large areas of zebra crossing color loss. It must be pointed out that the reconstruction of the zebra crossing area using global extraction and the proposed method is more complete, and the edge color distribution of the zebra crossing using the proposed method is also more accurate.

Quantitative evaluation: The comparative results of the indicators in Table 5 demonstrate that the proposed global feature extraction, local feature extraction, and spatial-frequency-domain discrimination have all improved the quality of image generation to varying degrees. Among them, the addition of a frequency-domain discriminator only leads to a slight decrease in the VIF metric. Overall, by using a dual-branch structure to selectively extract relevant features and calculate the corresponding saliency contrast loss, the learning process is constrained to maintain balance and consistency in the structure of the reconstructed image content. The dual-branch feature extraction and spatial-frequency-domain discrimination ensure the best quality and lowest distortion in the colorization results of infrared images, with the overall structure being closer to that of the original visible light images.

5. Conclusions

Using infrared image colorization methods to imbue infrared images with color information can reduce driving risks in harsh environments. This study mainly investigates the method of infrared image colorization, proposes and implements two innovative technologies, and verifies its reliability and practicability on the edge computing platform. Firstly, a dual-branch feature extraction network is proposed to address the issues of content consistency and structural preservation during the colorization of infrared images. This proposal combines GAN with Transformer and CNN dual-branch feature extractors to capture long-range features while retaining the original texture details. By introducing a global and local self-supervised saliency query module, the proposed method compares the losses between global and local self-supervision to ensure that the content and structure of the generated image are consistent with the original image, thereby improving the quality and stability of image colorization. Secondly, this study proposes a method that employs spatial- and frequency-domain discrimination strategies, addressing the issues of undersaturated and inaccurate coloring in image edge areas as well as the detail loss caused by deep networks during colorization tasks. The spatial discriminator refines color distribution, while the frequency-domain discriminator extracts key features and preserves original details through multi-scale geometric analysis transformation, making the generated color images more realistic and natural. Thirdly, an image colorization processing system has been established based on the Orin edge computing platform, dual-spectrum intelligent cameras, and monitors, completing hardware connections and testing, thereby verifying the feasibility and practicality of the proposed method.

Despite the above research findings, the proposed method in this study can be further optimized and improved. Although the proposed method based on a dual-branch feature extraction network is effective, the use of Transformer makes the overall structure complicated and increases the computation load, resulting in a slower training and inference speed. It is recommended to adopt lightweight technology and knowledge distillation to ensure effectiveness while reducing the model complexity and computation pressure. In addition, the method of combining spatial- and frequency-domain discrimination strategies has improved the authenticity of coloring and detail preservation, but the object edge lines generated in the image are still unstable. Our future research interest will be focusing on adding additional loss functions to the frequency-domain discriminator to enhance its constraint capability on the generator, thereby further improving the quality and stability of the image. Furthermore, employing multi-band infrared images can enable a rough separation of different types of surface materials within a scene, thereby allowing for better color restoration. This is an area of work that we intend to pursue in the next phase.