Self-Supervised Infrared Video Super-Resolution Based on Deformable Convolution

Abstract

Infrared video often encounters low resolution, which makes it difficult to perform the target detection and recognition task. Super-resolution (SR) is an effective technology to enhance the resolution of infrared video. However, the existing SR method of infrared image is basically a single image SR, which restricts the performance of SR due to ignoring the strong inter-frame correlation in video. We propose a self-supervised SR method for infrared video that can estimate the blur kernel and generate paired data from raw low-resolution infrared video itself, without the need for additional high-resolution videos for supervision. Furthermore, to overcome the limitations of optical flow prediction in handling complex motion, a deformable convolutional network is introduced to adaptively learn motion information to capture more accurate, tiny motion changes between adjacent images in an infrared video. Experimental results show that the proposed method can achieve an outstanding performance of restored image in both visual effect and quantitative metrics.

1. Introduction

Infrared imaging systems capture infrared radiation signals that are imperceptible to the human eye and convert them into visually recognizable image information through photoelectric conversion and processing. According to particularity and imaging mechanism of infrared radiation, infrared imaging technology has many advantages such as long imaging distance, strong anti-interference ability to penetrate smoke and haze, and working all weather, which have resulted in its extensive utilization across more application fields including power industry, construction industry, medical industry, petrochemical industry, port security, security and military. Despite rapid advancements in infrared imaging technology, infrared images often exhibit lower spatial resolution when compared to visible light images. SR restoration is an image processing technique for reconstructing high-resolution (HR) images from low-resolution (LR) images. In a traditional digital image, resolution is typically represented by the number of pixels in an image. Higher resolution implies more pixels, allowing for the display of greater detail and clarity. SR technology aims to recover missing details from LR images by utilizing information within the image or correlations between adjacent images. At present, the existing SR methods for infrared images are basically single-image SR (SISR), and most works on video SR are designed for video.

For SISR of infrared images, the methods based on neural networks have been explored by some researchers. An SISR technique based on multiscale saliency detection and residuals learned by a deep convolutional neural network (CNN) in the wavelet domain was proposed in [1], in which the incorporation of multiscale salient regions in sub-band images helps to minimize blurring artifacts caused by downsampling effects. In order to achieve a lightweight network, an iterative error reconstruction network based on efficient dense connections for infrared image SR is designed in [2], which achieves better performance using fewer parameters. Zewei He et al. [3] presented a cascaded architecture of deep neural networks with multiple receptive fields to increase the spatial resolution of infrared images by a large scale factor, achieving improved reconstruction accuracy with significantly fewer parameters. Aiming at not depending on paired datasets, Jing Han et al. [4] proposed a novel unsupervised infrared SR method based on the photoelectric characteristics of infrared images, establishing a fully unpaired generative adversarial framework to enrich image details without using high-resolution ground truth.

For the SR of visible light video, the goal of which is to infer the latent HR video from given LR frames. In recent years, people have gradually realized that inter-frame information has been overlooked in video SR. In fact, the information contained in the same object in adjacent frames of images has a tiny difference. If we can use information between adjacent frames to do SR restoration for video, it is definitely better than single-frame image SR theoretically. Therefore, restoration models based on the SR for static images [5,6,7,8,9,10], SR technology is expanding into the realm of video. In recent years, most methods of video SR are based on CNN. A video SR framework based on a CNN, which uses multiple LR frames as input to reconstruct one HR output frame, is introduced in [11]. Renjie Liao et al. [12] propose a new direction for fast video super-resolution via an SR draft ensemble that considers contextual information provided from external data for super-resolution. Jose Caballero et al. [13] introduce spatial-temporal sub-pixel convolution networks that effectively exploit temporal redundancies and improve reconstruction accuracy while maintaining real-time speed. Video SR typically extracts motion information, such as optical flow, to model interframe correlation, but it incurs high computational costs. Considering that recurrent neural networks can effectively model long-term correlation of video sequences, a fully convolutional recurrent neural network is proposed in [14] to achieve super-resolution restoration with good performance for complex motion videos with low computational complexity. In order to address the challenges of aligning multiple frames with large motions and effectively fusing different frames with diverse motion and blur, EDvR [15] devises a Pyramid, Cascading and Deformable (PCD) alignment module, in which frame alignment is done at feature level using deformable convolutions in a coarse-to-fine manner and a Temporal and Spatial Attention (TSA) fusion module, in which attention is applied both temporally and spatially to emphasize important features for subsequent restoration. TDAN [16] overcomes artifacts in warped supporting frames caused by inaccurate flow and image-level warping strategy and adaptively aligns the reference frame and each supporting frame at the feature level without computing optical flow. Yi Xiao et al. [17] propose to aggregate sharp information in severely degraded satellite videos with progressive temporal compensation, which exploits multiscale deformable convolution and deformable attention to explore sharper and cleaner clues by considering the blur level of pixels. Motion compensation and bidirectional residual convolutional network are combined in [18] to model spatial and temporal non-linear mappings, which is the first time motion compensated frames are regarded as inputs of recurrent convolutional nets for video SR, and recurrent convolutional layers are utilized to reconstruct residual images rather than output images in video SR. A method is proposed in [19] for high-quality and efficient video resolution upscaling tasks, which leverages spatial-temporal information to accurately divide video into chunks to keep the number of chunks and the model size to a minimum. The existing super-resolution methods usually impose optical flow between neighboring frames for temporal alignment. However, the estimation of accurate optical flow is hard and requires greater computation. GSTAN [20] effectively incorporates spatial-temporal information in a hierarchical way to address this problem. An end-to-end spatial-temporal video super-resolution network, which is chiefly composed of cross-frame transformers instead of traditional convolutions and a multi-level residual reconstruction module, which can make full use of maximum similarity and similarity coefficient matrices obtained by cross-frame transformers, is proposed in [21] to reconstruct high-resolution and frame-rate results from coarse to fine. To alleviate the problems of high computational costs and inefficient long-range information usage, a Bidirectional Recurrence Network (BRN) with an optical-flow-reuse strategy [22] is proposed to better use temporal knowledge from long-range neighboring frames for high-efficiency reconstruction. Yuming Jiang et al. [23] propose C2-Matching and design a dynamic aggregation module to address the potential misalignment issue between input images and reference images. Video SR methods usually require an accurate estimation of optical flow to wrap neighboring frames for temporal alignment, which is quite difficult. Therefore, an end-to-end deep convolutional network that dynamically generates spatially adaptive filters for alignment is proposed in [24] to avoid generating explicit motion compensation and utilize spatial-temporal adaptive filters to achieve the operation of alignment. Furthermore, TCNet [25] focuses on the temporal consistency of inter-frames and designs a spatial-temporal stability module to learn self-alignment from inter-frames to enhance the consistency of reconstructed videos. The aforementioned video SR methods based on CNN achieve significant performance. However, they are supervised learning approaches and depend on a dataset of HR-LR pairs to train a neural network, which is usually unavailable, especially in real-world applications. To address this problem, Haoran Bai et al. [26] propose a self-supervised learning method to solve the blind video SR problem, which simultaneously estimates blur kernels and HR videos from LR videos and introduces an optical flow estimation module to exploit information from adjacent frames for better HR video restoration.

The abovementioned methods for video SR are primarily designed for visible light video. Except for a lack of color, infrared videos also have characteristics of unclear texture details and unsharp edges compared to visible light videos. Therefore, the method of SR for visible light video may not work well for infrared video. In this paper, we propose an SR method for infrared video, which introduces a self-supervised structure that enables the estimation of blur kernels and generation of paired data directly from original LR infrared videos without the need for additional HR video supervision. Furthermore, it incorporates a deformable convolution network to adaptively learn motion information, allowing for a more precise capture of subtle motion variations, which circumvents complex processes and potential estimation errors associated with optical flow estimation.

The main contributions of this paper are as follows:

(1)

We propose a self-supervised infrared video SR method. We estimate the potential HR infrared video only from low-quality LR infrared video data, rather than relying on datasets of high-low resolution video pairs, which are often not available in practical applications.

(2)

We employ a deformable convolutional network to directly learn alignment and motion information between adjacent infrared video frames. Deformable convolution allows for local deformations to be learned for each pixel, enabling convolutional kernels to adaptively correspond to the shape and structure of the target, which can better preserve fine-grained details in the image, especially in cases of small motion displacement, as is often encountered in infrared videos.

(3)

Experimental results demonstrate that this proposed method can obtain high-quality HR infrared videos and perform better than comparison methods. This proposed method can be applied in video SR of various infrared imaging systems, especially for infrared imaging systems used for weak and small target detection and recognition.

2. Proposed Method

This overall framework of the proposed infrared video SR method consists of two branches. The main branch is used for estimating the blur kernel and potential HR frames. It employs a deformable convolutional network to learn motion information between adjacent frames and align frames. The auxiliary branch utilizes auxiliary paired data, which is generated based on LR input frames, and estimates the blur kernel from the main branch to constrain the training of the deformable convolution network and the clear frame reconstruction network. As shown in Figure 1, this overall framework comprises several components, including Blur Kernel Estimation Network N_k, Deformable Convolution Alignment Network N_w (which includes Feature Extraction Network N_e), and Clear Frame Reconstruction Network N_r.

2.1. Blur Kernel Estimation Network N_k

The blur kernel K_i can be estimated from the LR input frames. The blur kernel estimation network is shown in Figure 2, which is a neural network that includes Residual Channel Attention Blocks (RCAB) [8] modules and fully connected layers. RCAB combines the Channel Attention (CA) mechanism with the residual concept.

Each RCAB module maintains the same spatial resolution throughout. Specifically, it takes a feature map of shape (B,H,W,C) as input and processes it through two consecutive convolutional layers with a ReLU activation in between, followed by a CA mechanism. The CA mechanism first applies global average pooling along spatial dimensions, resulting in a tensor of shape (B,1,1,C), which captures global context for each channel. This pooled feature then passes through two $1\times 1$ convolutional layers with an intermediate ReLU and final sigmoid activation to produce channel-wise attention weights, also of shape (B,1,1,C). These weights are multiplied element-wise with the input feature map to produce an attention-modulated output (B,H,W,C), which is then added to the original input via a skip connection. As all intermediate feature maps maintain the same shape, no dimension adjustment is needed.

We clarify that in Figure 2, the skip connection is only used within the RCAB modules, where spatial dimensions remain unchanged, despite the use of pooling in the channel attention branch. The global average pooling only operates on spatial dimensions and compresses the feature map to (1,1) per channel without altering the feature dimensions required for residual addition. Therefore, all skip connections operate between tensors of identical shapes, ensuring compatibility throughout the network.

The blur kernel can be calculated by

$$K_i=N_k(C\{L_{i-N},\dots ,L_i,\dots ,L_{i+N}\})$$

(1)

where N_k represents the blur kernel estimation network, {L_i−N, …, L_i, …, L_i+N} represents LR input frames, and C represents the concatenation operation performed sequentially.

2.2. Deformable Convolutional Alignment Network N_w

In video super-resolution restoration, effectively utilizing information from neighboring frames is a critical problem. It is common to use alignment methods to integrate information from adjacent frames. Many optical flow estimation networks, such as PWC-Net [27], can accurately estimate optical flow between frames. Once frame-to-frame optical flow is available, a warp matrix can be constructed to compensate for the motion of objects in neighboring frames and align them with the reference frame [27]. Finally, aligned features are used for high-resolution frame restoration. In this paper, deformable convolution is employed to directly learn motion information, replacing the process of alignment based on optical flow calculations. Deformable convolution allows for local deformation at each pixel, enabling the convolution kernel to adaptively correspond to the shape and structure of the target, which can better preserve fine-grained details in the image, especially for infrared videos with small motion displacements. It avoids the complexity of optical flow estimation and the potential introduction of estimation errors. The deformable convolution alignment network is illustrated in Figure 3.

The features of LR input frames are initially extracted using a feature extraction network, as shown in Figure 4. Convolutional layers and five residual blocks are employed to extract features at scale p1 for each frame. Additionally, two convolutional layers are used to extract features at scales p2 and p3. This results in obtaining features at three different scales. The features at three scales can be calculated as follows:

$${f^{p1}}_j ,{f^{p2}}_j ,{f^{p3}}_j=N_e\left(L_j\right), j=i-N,\dots ,i,\dots ,i+N$$

(2)

Furthermore, utilizing deformable convolution and features at three different scales ${\{{f^{p1}}_j ,{f^{p2}}_j ,{f^{p3}}_j{\}}^{i+N}}_{j=i-N}$, we obtain the aligned feature ${\left\{w_{j\to i}\right\}}_{j=i-N}^{i+N}$ of the LR input neighboring frames with reference frame. Specifically, at scale p3, we estimate offsets, which are denoted as offset3 using ${f^{p3}}_i$ and ${f^{p3}}_j$, and the calculation formula is as follows:

$$offset3=Conv1(C\left\{f_i^{p3} , f_j^{p3} \right\})$$

(3)

where C represents the concatenation operation performed sequentially, Conv1 represents a general function composed of several convolution layers. The deformable convolution on offset3 and ${f^{p3}}_j$ is used to obtain aligned features ${w^{p3}}_{j\to i}$ at p3 scale, which is given by:

$$w_{j\to i}^{p3}=Conv2\left(Dconv\left( f_j^{p3} , offset3\right)\right)$$

(4)

where Conv2 represents a general function composed of several convolution layers, and Dconv represents deformable convolution.

$f_i^{p2}$ and $f_j^{p2}$ are concatenated and input together with the 2X up-sampled bilinear interpolation result with offset3 into a convolutional layer to obtain offset2. Then offset2 is convolved with $f_j^{p2}$ using deformable convolution, followed by inputting aligned features $w_{j\to i}^{p3}$ at scale p3 into a convolution layer to obtain aligned features $w_{j\to i}^{p2}$ at scale p2. Similarly, the offset1 and aligned features $w_{j\to i}^{p1}$ can be obtained at scale p1. The calculation formulas are as follows:

$$offset2=Conv1\left(C\{f_i^{p2},f_j^{p2}\},offset{2}^{\uparrow 2}\right)$$

(5)

$$w_{j\to i}^{p2}=Conv2\left(Dconv(f_j^{p2},offset2),{(w_{j\to i}^{p3})}^{\uparrow 2}\right)$$

(6)

$$offset1=Conv1\left(C\{f_i^{p1},f_j^{p1}\},offset{2}^{\uparrow 2}\right)$$

(7)

$$w_{j\to i}^{p1}=Conv2\left(Dconv(f_j^{p1},offset1),{(w_{j\to i}^{p2})}^{\uparrow 2}\right)$$

(8)

Subsequently, $f_i^{p1}$ and aligned features $w_{j\to i}^{p1}$ are concatenated at scale p1, and into a convolutional layer to obtain offset. Finally, offset is convolved with $w_{j\to i}^{p1}$ using deformable convolution to obtain the final aligned features $w_{j\to i}$. The calculation formulas are as follows:

$$offset=Conv1(C\left\{f_i^{p1} ,{ \mathrm{w}}_{j\to i}^{p1} \right\})$$

(9)

$$w_{j\to i}=Conv2\left(Dconv(w_{j\to i}^{p1},offset)\right)$$

(10)

The above operations are repeated to obtain aligned features ${\left\{w_{j\to i}\right\}}_{j=i-N}^{i+N}$ of all neighboring frames towards the reference frame.

2.3. Clear Frame Reconstruction Network N_r

The clear frame reconstruction network is shown in Figure 5, in which aligned features ${\left\{w_{j\to i}\right\}}_{j=i-N}^{i+N}$ are concatenated in sequence. Feature extraction is performed using a single convolutional layer and 20 residual blocks [28], followed by up-sampling using the sub-pixel convolution method [29]. Finally, the potential HR frame $P_i$ is obtained by overlaying it with the LR input reference frame $L_i$, which has been up-sampled using bicubic interpolation. $P_i$ is obtained by:

$$P_i=N_r\left(C\{w_{i-N\to i},\dots ,w_{i+N\to i}\}\right)+L_i^{\uparrow 4}$$

(11)

where $L_i^{\uparrow 4}$ represents LR input reference frame up-sampled by a factor of four using bicubic interpolation, C represents concatenation operation performed sequentially, $N_r$ represents clear frame reconstruction Network, and $p_i$ represents final reconstruction result.

2.4. Loss Function

The loss function L total consists of two parts: the main branch $loss L_{main}$ and the auxiliary branch loss $L_{auxiliary}$.

$$L_{main}=\rho (D{K}_i{p}_i-L_i)$$

(12)

$$\begin{array}{l}{L}_{auxiliary}=\rho \left((N_r\left(C[w_{i-N\to i},\dots ,w_{i+N\to i}]\right)+L_i^{\uparrow 4})-L_i\right)\end{array}$$

(13)

$$L_{total}=\lambda L_{main}+\gamma L_{auxiliary}$$

(14)

where $\rho (·)$ denotes $L_2-norm,$ D and $K_i$ represent down-sampling and blurring operations, respectively, $p_i$ is a potential HR frame, $L_i^{\prime }$ is an auxiliary LR sequence reference frame, $L_i$ is an LR input sequence reference frame, $\lambda$ and $\gamma$ have values equal to 1.

3. Experimental Results and Analysis

In the experiments, CPU used is an Intel Core i5-12490F @ 3.00 GHz with 32 GB of memory, and graphics processor is an NVIDIA RTX 3060. All deep learning experiments are implemented using the PyTorch framework. The installed packages in the environment include Python 3.10, NumPy 1.24.3, and OpenCV 4.7.0, etc. To accelerate training and computation, the platform utilizes CUDA 11.0 and CUDNN 8.0.3.

3.1. Datasets

The infrared video dataset used in this paper consists of two parts: one is obtained through an infrared dynamic scene simulation system, and the other is obtained through an infrared image capturing and processing system. The infrared dynamic scene simulation system can simulate various infrared images of infrared point targets and area targets. The infrared image capture and processing system can capture and process infrared targets and create images of them. The infrared images used in our experiments include 25 videos with infrared targets such as aircraft and missiles. Among these videos, 20 videos are used as training sets, while test sets are composed of the remaining 5 videos and a publicly available infrared video obtained from the Internet. When obtaining lower-resolution video image data for test sets, a bicubic down-sampling method is used. In the training phase, the initial learning rate is set to 2.5 × 10⁻⁵, and the optimizer used is Adam [30] with parameter settings ${\beta }_1=0.9,{\beta }_2=0.999,\epsilon ={10}^{-8}$. The learning rate is reduced by half every 20 epochs, with a total of 60 epochs of training.

3.2. Results and Analysis

The performance of the proposed method is compared with traditional bicubic interpolation and a self-supervised video SR method in [27]. The analysis is conducted by quantifying results using subjective and objective evaluation metrics.

3.2.1. Subjective Results

The example frames extracted from the original infrared video are shown in Figure 6, in which the middle frame is used as a reference frame.

The LR example frames after performing down-sampling by a factor of 4× on the above original infrared frames are shown in Figure 7.

The SR reconstruction of middle frame of LR infrared frames in Figure 7 is performed using bicubic interpolation, method in [26], and proposed method in this paper, respectively. The results of the SR reconstruction are shown in Figure 8.

From Figure 8, it can be seen that bicubic interpolation restoration is relatively blurry, with unclear edge details and an inability to present high-frequency details clearly. It performs the worst in subjective evaluation metrics. Although the method in [26] has restored some high-frequency details compared to bicubic interpolation, it still suffers from unclear edge details and relatively blurry images. The images reconstructed by the proposed method in this paper are clearer compared to the other two methods.

From Figure 6, Figure 7 and Figure 8, it can be seen that the original images have strong noise. It shows that the original images have poor anti-noise ability, and even noise amplification occurs locally. At the same time, the outline of the target is also fuzzy. From Figure 6, Figure 7 and Figure 8, it can be seen that the Self-Supervised Super Resolution Restoration algorithm (SSRR) has a strong ability to suppress noise. From the restoration results, the background noise is filtered out. Overall restoration performance is good. It has a better restoration effect. The noise in the high-resolution restored image cannot be perceived from subjective observation.

3.2.2. Objective Results

The commonly used objective evaluation metrics, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measurement (SSIM), are used to evaluate the reconstruction effects of each method. The PSNR is calculated by:

$$MSE={\displaystyle \frac{1}{N}}\times {\displaystyle \sum _{i=1}^N[{(I(i)-M(i))}^2]}$$

(15)

$$PSNR=10\times {\log }_{10}^{(MA{X}^2/MSE)}$$

(16)

MSE is mean squared error, representing differences in pixel values at corresponding positions between two images, N is the total number of pixels, $I(i)$ is the pixel value in the original image, $M(i)$ is pixel value in SR reconstructed image, and MAX is the maximum pixel value of the image, which is typically set to 255. A higher PSNR value indicates that differences between the SR restored image and the original image are smaller, indicating better HR image quality. However, PSNR primarily focuses on the difference between pixels and is less sensitive to perceptual loss of details as perceived by human eyes.

SSIM not only considers the difference between pixels but also takes into account structural information and human perceptual characteristics of an image. Assuming that two input images are x and y, the calculation of SSIM is as follows:

$$SSI{M}_{(x,y)}={\displaystyle \frac{(2u_x{u}_y+c_1)({\sigma }_{xy}+c_2)}{(u_x^2+u_y^2+c_1)({\sigma }_x^2+{\sigma }_y^2+{\mathrm{c}}_2)}}$$

(17)

where $u_x$ and $u_y$ represent means of x and y, ${\sigma }_x$ and ${\sigma }_y$ represent standard deviations of x and y, and ${\sigma }_{xy}$ represents covariance between x and y. $c_1$ and $c_2$ are constants to avoid division by zero, typically taken as small positive numbers. SSIM value ranges from 0 to 1, with values closer to 1 indicating a higher structural similarity between the SR reconstructed image and the original image, which indicates better HR image quality. For each video, the average PSNR and SSIM of all frames are calculated. The experimental results are shown in

Table 1. Quantitative comparison results of SR using different methods.

Method	Bicubic	Method in [26]	Ours
PSNR	26.935	28.233	29.890
SSIM	0.826	0.857	0.881
Time (s)	0.4	4.2	2.7

. The processing time of the proposed method is lower than that of the other method. The other methods are poor in noise suppression and edge detail retention, and the processing time is longer.

4. Conclusions

In this paper, we propose an infrared video SR method based on self-supervision and deformable convolution. It obtains a blur kernel from the LR input sequence itself and ultimately generates an image of the same size as the LR input sequence through the main branch and the auxiliary branch. A loss function is constructed between LR input frames to achieve self-supervision. In addition, deformable convolution is used to directly learn motion information, avoiding the complex process of optical flow estimation and potential estimation errors, which can better adapt to complex motion and deformation in infrared videos, providing a new direction for infrared video SR restoration methods. It can be found from experimental results that the proposed method in this paper can achieve satisfactory results without the need for paired HR-LR video datasets, and it is particularly effective in enhancing dim small targets, which has important practical significance for the detection and recognition of dim small targets. Although the proposed method achieves promising performance on infrared video super-resolution, there is still room for further improvement in terms of computational efficiency and generalization. In future work, we will explore lightweight backbone designs, such as employing depthwise separable convolutions or dynamic convolution modules, to reduce computational overhead. We also plan to incorporate model compression techniques, including pruning and quantization, to accelerate inference and reduce memory consumption. In addition, replacing the deformable alignment module with more efficient motion compensation strategies, such as attention-guided warping and object tracking [30] may help strike a better balance between performance and speed. Finally, we intend to extend our framework to related video tasks, such as video denoising and temporal super-resolution, to evaluate its adaptability in broader scenarios.