Infrared Image Super-Resolution via Progressive Compact Distillation Network

Abstract

Deep convolutional neural networks are capable of achieving remarkable performance in single-image super-resolution (SISR). However, due to the weak availability of infrared images, heavy network architectures for insufficient infrared images are confronted by excessive parameters and computational complexity. To address these issues, we propose a lightweight progressive compact distillation network (PCDN) with a transfer learning strategy to achieve infrared image super-resolution reconstruction with a few samples. We design a progressive feature residual distillation (PFDB) block to efficiently refine hierarchical features, and parallel dilation convolutions are utilized to expand PFDB’s receptive field, thereby maximizing the characterization power of marginal features and minimizing the network parameters. Moreover, the bil-global connection mechanism and the difference calculation algorithm between two adjacent PFDBs are proposed to accelerate the network convergence and extract the high-frequency information, respectively. Furthermore, we introduce transfer learning to fine-tune network weights with few-shot infrared images to obtain infrared image mapping information. Experimental results suggest the effectiveness and superiority of the proposed framework with low computational load in infrared image super-resolution. Notably, our PCDN outperforms existing methods on two public datasets for both ×2 and ×4 with parameters less than 240 k, proving its efficient and excellent reconstruction performance.

1. Introduction

Infrared imaging plays a significant role in military target detection, video surveillance [1], biomedical science [2], and other areas [3,4,5]. However, infrared images generally suffer from low contrast and blurred details. To tackle this problem, many super-resolution (SR) methods have been proposed. Single-image super-resolution (SISR) is where clear high-resolution (HR) images can be reconstructed from a blurry single-frame low-resolution (LR) image based on the identity mapping function or imaging model through prior information. Dong et al. [6] pioneered the three-layer convolutional network into SR to learn nonlinear LR–HR mapping. Subsequently, SR deeper-oriented approaches have received significant attention. A 20-layer model named VDSR was proposed by Kim [7], which was inspired by the proposal of the residual network [8] and the VGG model [9]. Furthermore, extensive models, EDSR [10] and RCAN [11] with 69-layer and 400-layer networks, respectively, have achieved state-of-the-art performance.

Although visible image SR techniques have displayed tremendous progress, with deeper network structure models possessing better nonlinear expression ability to present superior image quality, there are a few limitations for their adoption in the infrared image domain. Firstly, these SR networks lead to a huge computational cost, require large computer memory resources, and are not suitable for real-world infrared detector devices. Designing fast, accurate, and lightweight infrared image SR models combined with infrared visual perception is a promising approach to mitigate this contradiction. Secondly, training such deep network models requires massive datasets of different scenes or targets as sample support, which hinders the assembly of infrared image datasets due to expensive optical sensors and restricted natural circumstances. The main alternative approach is to utilize transfer learning to address insufficient training dataset resources in the infrared domain.

As lightweight networks, DRCN [12] and DRRN [13] utilized recursive networks to decrease redundant parameters. CARN-M [14] introduced an efficient and lightweight cascaded residual network with group convolution for mobile devices; however, this was at the expense of a serious loss of PSNR with the stacking of group convolution. Hui et al. [15] designed the information multi-distillation network (IMDN), which divides the preceding features into two segments, whereby one part is maintained and the other is further processed. MSAR-Net [16] proposed a multiscale attention residual block for feature refinement and an up–down sampling projection block for edge refinement, further improving the efficiency of extracting high-frequency information.

For limited training datasets in the target field, transfer learning [17] can effectively apply learned knowledge in a previous field to a novel domain. Shahin [18] et al. proposed the transfer deep learning approach for examination tasks and designed the lightweight DCRN to estimate building orientation angle. Zhang [19] et al. improved the accuracy of spectrum classification and enhanced the robustness of the model using a multistep training strategy applying different datasets. Taking advantage of transfer learning, based on the visible image in the first stage, the second stage employs few-shot infrared image samples to update network weights.

In this paper, we design a progressive compact distillation network (PCDN) integrated with few-shot transfer learning using easily available visible images and few-shot infrared image samples. The goal of the proposed PCDN framework is to efficiently restore the SR infrared images in real time. The core progressive feature distillation block (PFDB) utilizes the combination of asymmetric convolution and parallel dilation convolution with the multi-receptive field, which can not only maximize the characterization power of marginal features but also minimize the network parameters. For adjacent PFDBs, the proposed PCDN investigates the difference calculation algorithm concentrating on the high-frequency information to provide structure priors. Moreover, we propose the bil-global connection mechanism to significantly lower the complexity of the network and accelerate the convergence. Secondly, PCDN applies transfer learning to reuse the features trained by the visible image dataset for the infrared domain. Specifically, the PCDN is trained by visible images to create an efficient ensemble mapping relationship to enhance the restoring performance. However, it is not ideal to employ the trained model to super-resolved infrared images since the mapping weights are obtained from the visual image instead of the infrared image. Consequently, we fine-tune the network weights based on the pretrained model through a small number of infrared images in the second step. Compared with existing transfer learning methods, the PCDN only employs 55 infrared images for the fine-tuning stage and achieves excellent performance. We discuss the experimental details in the ablation experiments and evaluate our method on two public infrared datasets to compare its performance with existing SR methods.

This paper is organized as follows: Section 2 reviews the state-of-the-art methods in the field of SR and transfer learning. Our proposed method is elaborated in Section 3. Results and a discussion are presented in Section 4 and Section 5, respectively.

2. Related Work

2.1. Lightweight CNNs for SR

The current CNN-based SR models achieve optimal performance regardless of negative effects caused by a deeper network or larger filter size. For instance, the EDSR of Lim et al. [10] stacked 69-layer networks with a sequence of residual blocks, and the RCAN designed by Zhang et al. [11] even exceeded 400 layers through residual in residual (RIR) to achieve state-of-the-art performance. However, such networks of excessive parameters with deeper structure and larger kernel size lead to huge calculation, limiting real-world applications. Hence, numerous researchers have designed strategies, such as pre-upsampling, global and local structure, and internal convolution operation.

Compared with the originally proposed SRCNN [6], FSRCNN [20] changed the pre-upsampling strategy that extracts features from the bicubic interpolated LR images to the post-upsampling strategy that directly takes the LR images as input, thereby greatly lowering the computational complexity, and it has been applied to almost all SISR tasks.

The global and local structure strategy focuses on transforming the connection among modules to enhance the sharing and utilization of information. The DRCN proposed by Kim et al. [12] is the premier model in SR used in a recursive manner to obtain more high-frequency feature spaces from the target pixels. CARN [14] implements a cascading mechanism at both the local and the global levels to aggregate the features from multiple layers. Hui et al. [15] designed the information multi-distillation network (IMDN) for better purifying each processing feature by explicitly splitting the preceding features into two segments, whereby one part is maintained and the other is further refined, streamlining the network parameters and boosting the reconstruction performance. RTAN [21] is a novel lightweight residual triplet attention module used to obtain the cross-dimensional attention weights of the features. LW-AWSRN [22] is a novel local fusion block consisting of adaptive weighted residual units and the local residual fusion unit to remove the redundancy scale branch.

From the perspective of the internal convolution operation, several researchers have made great progress. InceptionV3 [23] factorized an n × n convolution to a 1 × n followed by an n × 1 convolution, reducing the parameters by nearly 33% with an identical receptive field. CARN-M [7] introduced group convolution based on the cascaded residual network. Group convolution separates the channel of the filter to slide over the input images, and depth-wise convolution [24] is used in the extreme group convolution to minimize the parameters. Nevertheless, as the stacking depth-wise convolution layers increases, the representative ability is drastically weakened due to the information fusion from different channel groups being hindered. Dilation convolution [25], with the same parameters as vanilla convolution, expands receptive field by filling holes in the filter window and generates abundant marginal feature by assembling multiscale dilation convolution.

2.2. Infrared Image SR and Transfer Learning

WGAN [26] employs a heterogeneous kernel-based residual block that enables multiscale extraction of infrared image features by combining convolutional kernels of different sizes. MRIDN [27] introduces effective multi-receptive field convolution groups based on a fixed number of the channels split to extract features of infrared images. Although these models reconstruct high-quality infrared images, they all rely on a large number of datasets as the foundation. However, certain domains make it difficult to assemble sample sets due to expensive equipment or restricted natural circumstances. Hence, transfer learning has emerged in many methods in various image processing fields. For instance, Moran [28] analyzed medical images through a pretrained network and successfully recovered digital periapical radiographs. Chen [29] et al. exploited different transfer learning implementations, i.e., pretrained deep learning models combined with support vector machine (SVM) and fine-tuning, in detecting unfavorable driving states. In the infrared image domain, PSRGAN [30] employs a multistage transfer learning strategy utilizing visible images and 100 infrared images to boost the restoration performance of infrared images. In this paper, the proposed PCDN only employs 55 infrared images to fine-tune the pretrained network and performs better compared to existing SR approaches.

3. Method

3.1. Overall Architecture of the Network

The overall framework of the proposed network, shown in Figure 1, is composed of four modules: (1) first feature extraction (FFE) convolution for collecting the robust features, (2) stack of progressive feature residual distillation (PFRD) modules for generating nonlinear mapping relationship, (3) difference calculation (DC) between PFRDs for exploring high-frequency information, and (4) reconstruction module with (upsampling pixel attention) U-PA blocks.

Firstly, we set the original LR image ($I_{LR}$) as the input of our network. The FFE was implemented using only a 3 × 3 convolution, followed by the Relu activation function to map $I_{LR}$ from the image space to feature space, expressed as follows:

$$x_0=h_{FFE}\left(I_{LR}\right),$$

(1)

where $h_{FFE}$ denotes the 3 × 3 convolution operation, and $x_0$ is the extracted shallow feature.

Then, several PFRDs were followed for deep feature nonlinear mapping to generate powerful feature representations. We denote the proposed the PFRD as $f_{PFDB}\left(·\right)$, given by

$$x_{pf}^n=f_{PFDB}^n\left(f_{PFRD}^{n-1}\left(\dots f_{PFRD}^0\left(x_0\right)\dots \right)\right),$$

(2)

where $x_{pf}^n$ is the output feature map of the n-th PFRD module. Then, the latter PFRD block subtracts the former counterpart for providing structure priors, which can be formulated as

$$x_{dc}^{n-1}=x_{pf}^n-x_{pf}^{n-1},$$

(3)

where $x_{dc}^i$ denotes the difference between two adjacent PFRD blocks. After passing through the last PFRDs and DCs, we concatenate all generated $x_{pf}^n$ and $x_{dc}^i$ to feed into the reconstruction module. This process is defined below.

$$I_{SR}=f_{ rec}\left(Concat(x_n+x_{dc}^{n-1}\right)+x_0)+f_{bil}\left(x_0\right),$$

(4)

where $f_{ rec}$ represents the U-PA reconstruction function, inspired by PAN [31], and $Concat$ denotes the concatenation operation along the channel dimension. In addition, we added the bil-global connection that accumulates the shallow feature $x_0$ and a global bilinear interpolation connection $f_{bil}$. $I_{SR}$ is the final output of the network.

3.2. Progressive Feature Residual Distillation

As depicted in Figure 1, our progressive feature residual distillation (PFRD) module is composed of several asymmetric residual blocks (ARBs) aimed at extracting deeper features, two parallel asymmetric convolutions with different dilation factors for expanding the receptive field, a few 1 × 1 convolutions on the distillation pipeline used to reduce the number of channels, and subsequent processing for the concatenation layer.

As for the ARB, it halves the two convolutions extracting features to compress the number of parameters compared with the vanilla residual block. Afterward, the single convolution is decoupled into consecutive 1 × 3, 3 × 1 convolutions to substantially diminish the parameters. As a consequence, the ARB can better leverage residual learning through stacking residual connections with an extremely modest number of parameters.

The PFDB first divides the previous feature into two components: the main refinement path and the branch distillation path. The bottom main path adopts three serial ARBs to better purify features, while the upper branch path is used to preserve coarse feature information using a 1 × 1 convolution filter behind each ARB. We take the insufficient depth spatial context into account since the ARB has only one 3 × 3 convolutional. Hence, two parallel dilation convolutions are complemented at the end of the main distillation branch to increase the receptive field and extract more marginal contextual information. This procedure can be described as follows:

$$\begin{gathered} \mathcal{F}=ReLU\left(Con{v}_{3\times 1}\left(Con{v}_{1\times 3}\left(x\right)\right)+x\right), \\ f_{refined}^1, f_{coarse}^1=\mathcal{F}\left(f_{in}\right), \mathcal{D}\left(f_{in}\right), \\ {f}_{refined}^2, f_{coarse}^2=\mathcal{F}\left(f_{refined}^1\right), \mathcal{D}\left(f_{coarse}^1\right), \\ {f}_{refined}^3, f_{coarse}^3=\mathcal{F}\left(f_{refined}^2\right), \mathcal{D}\left(f_{coarse}^2\right), \\ {f}_{refined}^{4\_1}, f_{refined}^{4\_2}={\mathcal{F}}_{Di{l}_2}\left(f_{refined}^3\right), {\mathcal{F}}_{Di{l}_3}\left(f_{coarse}^3\right), \\ {f}_{out}=Concat\left(f_{coarse}^1, f_{coarse}^2, f_{coarse}^3, f_{refined}^{4_1}, f_{refined}^{4_2}\right), \end{gathered}$$

(5)

where $\mathcal{F}$ and $\mathcal{D}$ represent the ARB and 1 × 1 convolution filter, and $f_{refined}^i$ and $f_{coarse}^i$ denote the output of i-th ARB block and i-th distillation layer, respectively. ${\mathcal{F}}_{Dil\_2}$ and ${\mathcal{F}}_{Dil\_3}$ represent the convolution filter in the ARB block with dilation factors equal to 2 and 3, respectively; $f_{out}$ is the aggregated output of all refined and distilled features.

After concatenating all features in PFRD, we exploit the channel shuffle mechanism, inspired by ShuffleNet [32], to strengthen the correlation and share the features among concatenated channels, followed by a 1 × 1 convolution aimed at reducing the dimension of the features. Furthermore, we adopt the ESA_D, inspired by ESA [33], which replaces the convolution group with dilation convolution, and we apply the depth-wise convolution as input to refine the key information of the feature with negligible parameters.

3.3. Difference Calculation and Bil-Global Connection Mechanism

After obtaining refinement features from the inside of the PFRD, we further make full use of feature correlations of adjacent PFRDs through difference calculation. From Figure 1, the latter PFRD block has more high-level information than its former counterpart, because more convolutional layers can extract more refined and more representative features. Hence, $f_{PFDB}^n$ subtracted from $f_{PFDB}^{n-1}$ can obtain high-frequency information and provide structure for post-processing image SR. On the other hand, we designed a bil-global connection mechanism to stabilize the network and boost reconstruction performance. This mechanism implements the global connection from the LR low-resolution image and the extracted shallow image by bilinear interpolation. The backpropagation signal can directly pass through these two global skip connections to make our network easier to train. The overall operations of the difference calculation and bil-global connection mechanism are illustrated in Equations (3) and (4), as well as Figure 1.

3.4. Loss Function

We choose the L1 loss function at the visible training stage and fine-tune the network with the L2 loss function at the transfer learning stage. Given a training set ${\left\{I_{HR}^i, I_{LR}^i\right\}}_{i=1}^N$ that has LR–HR pairs, the goal of PCDN is to minimize the following loss functions:

$$L_1\left(\theta \right)=\frac{1}{N}{\displaystyle \sum }_{i=1}^N \Vert H_{PCDN}\left(I_{LR}^i\right) —I_{HR}^i{\Vert }_1,$$

(6)

$$L_2\left(\theta \right)=\frac{1}{N}{\displaystyle \sum }_{i=1}^N {(H_{PCDN}\left(I_{LR}^i\right) —I_{HR}^i)}^2,$$

(7)

where $\theta$ indicates the training parameters of the PCDN, $H_{PCDN}$ represents our proposed model, and $I_{LR}$ and $I_{HR}$ represent the LR image and its corresponding ground-truth HR image in the respective dataset.

4. Results

4.1. Implementation Details

In our proposed method, we used the recently popular dataset DIV2K [34] as the visible image dataset to train our models. The DIV2K dataset contains 800 high-quality RGB training images. For infrared images, we adopted the CVC-09: FIR Sequence Pedestrian dataset [35]. We extracted 55 images from this dataset as the infrared image training datasets named IR55. We employed two datasets for testing, defined as result-A and result-C, which consisted of 22 images obtained by fusing the infrared images and visible light images utilizing the approaches proposed in [36,37], respectively. We used the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) to measure the quality of the reconstructed images. All values were converted from RGB channels to the Y channel of YCbCr for evaluation.

We generated the training LR images by downsampling HR images with scaling factors (×2 and ×4) using bicubic interpolation in MATLAB. We trained the model using visible DIV2K datasets and fine-tuned it using IR55 at the visible stage and transfer learning stage. At both stages, we used the 128 × 128 RGB input patches from the LR image for training and trained our model with the ADAM optimizer by setting β₁ = 0.9, β₂ = 0.999, and ε = 10⁻⁸. The learning rate was initialized as 5 × 10⁻⁴ and halved every 2 × 10⁵ minibatch updates at the first stage, whereas it was set as 5 × 10⁻⁵ and halved every 20 minibatch updates at the second stage. We set the batch size to 32 at both stages and set the number of epochs to 400 at the visible training stage for full training. However, since the pretrained network tended to converge, the epoch was set to 50 at the transfer learning stage. The networks were implemented utilizing the PyTorch framework with NVIDIA TITAN X (Pascal) GPUs.

4.2. Quantitative and Qualitative Results

We compared our approach quantitatively with popular SR methods including SRMD [38], DPSR [39], EDSR [10], SPSR [40], IMDN [15], and PSRGAN [24]. Results of PSNR and SSIM values are presented in

Table 1. Average PSNR/SSIM for scale factors of 2 and 4 on datasets result-A and result-C.

Test Datasets	Scale	SRMD [38]	DPSR [39]	EDSR [10]	SPSR [40]	IMDN [15]	PSRGAN [30]	Ours
result-A	×4	11.90/0.2860	32.45/0.8181	27.89/0.7076	32.46/0.7839	32.56/0.8239	33.13/0.8282	34.95/0.8647
result-A	×2	23.21/0.4193	37.01/0.9303	31.94/0.8106	36.41/0.8961	37.16/0.9343	37.48/0.9229	39.31/0.9441
result-C	×4	12.52/0.4193	33.12/0.8377	28.41/0.7332	33.14/0.8056	33.20/0.8435	33.86/0.8466	35.62/0.8808
result-C	×2	22.92/0.7605	37.91/0.9425	32.46/0.8294	37.23/0.9129	38.06/0.9465	38.52/0.9363	40.29/0.9558

. It can be found that PCDN achieved the best performance and outperformed these methods by a large margin on ×2 and ×4 scaling factors in both testing datasets. It is worth noting that the performance of PSRGAN was inferior to that of our network. Firstly, PCDN only uses 55 infrared images for transfer learning while PSRGAN used 100, which means that the former uses less sample space. Secondly, PCDN has fewer parameters (227.403 K) than PSRGAN (312.7 K). In addition, in terms of reconstruction performance, the proposed PDCN outperformed the PSRGAN for both result-A and result-C on all scales, being 1.76 dB and 0.0342 higher than PSRGAN in terms of PSNR and SSIM, respectively, for result-C ×4. In conclusion, our PCDN achieved better infrared image reconstruction performance with fewer parameters and less infrared sample space, highlighting the powerful superiority of our network. Therefore, it is clear from Table 1 that the proposed PCDN performs well when compared with existing methods.

We evaluated the visual results of different models to further illustrate the effectiveness of our method. The visual comparisons on the scale ×2 and ×4 result-A and result-C are visually depicted in Figure 2. It can be seen from the zoomed in part of the eaves for image_18 in result-A × 4 that the PCDN could reconstruct more realistic textures. For image_7 in result-C × 2, the softer and smoother stripes and line patterns were better recovered by PCDN than other methods. Overall, the capability of the proposed network to synthesize greater detail and more accurate structures was proven for infrared image super-resolution.

5. Discussion

5.1. Model Analysis

In many typical factors used to assess a lightweight model, the essential and vital standard of measurement involves the network parameters. Therefore, we visualize a cost-effectiveness analysis between PSNR and model size for the result-C ×4 dataset in Figure 3. Our network is extremely lightweight, and its parameters are only 227.403 K when ×2 and 235.063 K when ×4. In addition, we also present a tradeoff analysis between performance and Multi-Adds for result-C ×2 in Figure 4. The Multi-Adds of PCDN was only 14.64 G and 22.02 G when ×2 and ×4, respectively, which is much lower than other methods. It can be observed that our PCDN achieved the optimal tradeoff between reconstruction performance and model size in contrast to the existing models.

5.2. Ablation Experiments

In order to demonstrate the effectiveness of transfer learning, we analyzed and compared the two training results utilizing the DIV2K visible dataset individually and with IR55 transfer learning. We trained 400 epochs for these ablation experiments. The experimental results prove that the performance of the PCDN was improved using transfer learning, as shown in

Table 2. Evaluation of performance in training using DIV2K visible dataset individually and with IR55 transfer learning.

Training Datasets		Test Datasets
DIV2K	IR55	result-A ×2	result-A ×4	result-C ×2	result-C ×4
√	×	39.2224/0.9447	34.9491/0.8645	40.2422/0.9552	35.5849/0.8805
√	√	39.3126/0.9441	34.9547/0.8647	40.2940/0.9558	35.6215/0.8808

. For result-A ×2, PCDN enhanced the PSNR by 0.09 dB and of the SSIM by 0.006 for result-C ×2. This shows that fine-tuning the pretrained network of transfer learning is beneficial to improve the performance of the model without introducing parameters.

Furthermore, we investigated the implementation of dilation convolution in PFRD. We compared and analyzed the impacts of using dilation convolution and common convolution at the end of the distillation feature extraction on the network performance. Figure 5 shows the training process of the calculated PSNR with 800 images in DIV2K as the training dataset and 100 images as the validation dataset. It can be observed that the PCDN with dilation convolution showed better performance. Moreover, the PSNR and SSIM on the infrared test datasets for result-A and result-C are presented in

Table 3. Investigations of common convolution and dilation convolution at the end of distillation feature extraction for two test datasets.

Block Composition	PSNR/SSIM
Test Datasets	Result-A ×2	Result-A ×4	Result-C ×2	Result-C ×4
PFRD_Common	39.16/0.9427	34.88/0.8634	40.13/0.9540	35.50/0.8799
PFRD_Dilation	39.31/0.9441	34.95/0.8647	40.29/0.9558	35.62/0.8808

.

Due to the superiority of dilation convolution in expanding the receptive field, we further explored whether the reconstruction performance would improve upon replacing all convolutions in the PFRD by dilation convolution. However, this caused performance to plummet. This was possibly due to the holes inserted between effective pixel values of filters when convolution introduced the dilation factor, which led to feature information being lost, especially in the underlying network containing more important details.

6. Conclusions

In this paper, we presented the progressive compact distillation network (PCDN) incorporating the lightweight information distillation network and transfer learning strategy. We designed the progressive feature residual distillation block (PFRD) containing asymmetric residual blocks, parallel asymmetric convolutions with different dilation factors for the feature extraction, and a subsequent processing module with adaptive recalibration for the weights of fusion features. In terms of structure design, we then utilized the difference calculation between adjacent PFRDs to obtain high-frequency information and the bil-global connection mechanism to keep the network stable. Moreover, we used the transfer learning strategy to fine-tune the PCDN with only 55 infrared images based on the pretrained network using visible image datasets. Comprehensive experiments illustrated that our proposed extremely lightweight method realizes optimal infrared image super-resolution performance with few-shot infrared images samples. In the future, we will further explore our approach to achieve better infrared image reconstruction with zero-data learning in the transfer learning domain.