Adaptive Frame Sampling and Feature Alignment for Multi-Frame Infrared Small Target Detection

Abstract

In recent years, infrared images have attracted widespread attention, due to their extensive application in low-visibility search and rescue, forest fire monitoring, ground target monitoring, and other fields. Infrared small target detection technology plays a vital role in these applications. Although there has been significant research over the years, accurately detecting infrared small targets in complex backgrounds remains a significant challenge. Multi-frame detection methods can significantly improve detection performance in these cases. However, current multi-frame methods face difficulties in balancing the number of input frames and detection speed, and cannot effectively handle the background motion caused by movement of the infrared camera. To address these issues, we propose an adaptive frame sampling method and a detection network aligned at the feature level. Our adaptive frame sampling method uses mutual information to measure motion changes between adjacent frames, construct a motion distribution, and sample frames with uniform motion based on the averaged motion distribution. Our detection network handles background motion by predicting a homography flow matrix that aligns features at the feature level. Extensive evaluation of all components showed that the proposed method can more effectively perform multi-frame infrared small target detection.

1. Introduction

In recent years, with advancements in science and technology, infrared images have found widespread use in low-visibility search and rescue, forest fire monitoring, ground target monitoring, and other fields [1,2,3,4]. Infrared small target detection (IRSTD) plays a crucial role in these applications, and numerous researchers have extensively studied this area for decades [5]. Most targets in IRSTD include ships, aircraft, and vehicles, with backgrounds of sky, ocean, and land [6]. Therefore, these targets often lack significant shape and texture, lack color information, and exhibit low contrast with the background in infrared images, and are often submerged in complex backgrounds [7]. For these reasons, accurately detecting the location of small targets in infrared images remains a very challenging problem [8].

Based on the use of infrared image frames, the current mainstream infrared small target detection methods can be divided into single-frame-based methods and multi-frame-based methods [9]. Single-frame-based detection methods mainly locate small infrared targets by separating background features from the target, and do not utilize the motion information of the targets. As shown in Figure 1, in scenarios where the target is embedded in a complex background and is small, occupying only a few pixels, the effectiveness of single-frame-based methods in distinguishing noise from the target is often limited.

By leveraging the temporal context, multi-frame-based methods can extract motion information from sequences [10]. As a result, multi-frame-based methods generally outperform single-frame-based methods, especially for complex backgrounds. The motion information helps build more robust infrared small target detection models [11]. This paper primarily discusses the difficulties and challenges faced by multi-frame-based infrared small target detection methods.

Multi-frame-based methods usually take the current frame $F_t$ and the previous k consecutive frames, i.e., $[F_{t-k},\dots ,F_{t-1},F_t]$, as input. While the detection accuracy of the model is proportional to the number of input frames within a certain range, increasing the number of input frames significantly increases the computational workload, resulting in longer detection times. This poses challenges in scenarios with strict real-time detection requirements and limited computing resources, such as high-speed airborne platforms, unmanned aerial vehicle (UAV) systems, and handheld embedded devices. These situations present significant constraints and limitations in terms of the computational efficiency of infrared small target detection. As shown in Figure 1, the information contained between adjacent frames is often redundant. Enhancing detection performance and computational efficiency without increasing the computational burden can be achieved by extracting N frames from a sequence of T consecutive frames. This approach allows for the incorporation of more motion and scene details, leading to improved detection performance, while maintaining a favorable computational cost ratio.

Most of the currently used methods sample frames from consecutive dense frames at random or fixed intervals. However, this fixed strategy sampling method has the following issues. First, the motion patterns of different dense frame sets vary, and the targets do not move at a uniform speed. A fixed strategy sampling method may lose important information in some sets. Second, this method may sample redundant frames in other sets. Another issue with multi-frame infrared small target detection is the interference caused by background displacement due to the motion of the infrared detection sensor. Recently, this problem has been addressed by pre-processing the input frames to align them with the perspective of the detection frame $F_t$ [12,13,14]. However, this approach heavily relies on the quality of image registration preprocessing. Improper registration can introduce significant interference into the detection process, leading to poor robustness.

To solve these issues, this paper proposes a frame sampling pre-processing method based on mutual information for multi-frame infrared small target detection. Additionally, a multi-frame infrared small target detection model is introduced, which performs alignment at the feature level. The designed frame sampling method based on mutual information quantifies the motion variation between consecutive frames by computing the mutual information of the input dense frames. By accumulating the motion changes from $f_{t-k}$ to $f_t$, frame sampling is adaptively performed. Moreover, the proposed multi-frame infrared small target detection model, which performs alignment at the feature map level, embeds the registration operation from the preprocessing stage into the detection model and the model is trained in an end-to-end manner, enhancing its robustness.

The main contributions of this paper are summarized as follows:

• We propose an adaptive frame sampling method based on mutual information for multi-frame infrared small target detection, to overcome the shortcomings of the fixed strategy frame sampling method.
• A multi-frame infrared small target detection model is proposed, which incorporates feature alignment by placing the registration operation at the feature map level. The model is trained in an end-to-end fashion, overcoming the limitations of manual registration in the preprocessing stage in terms of robustness.
• A comprehensive set of experiments were conducted on a mobile infrared small target dataset to evaluate the effectiveness of the proposed method. The results provide strong evidence supporting the efficacy of the proposed approach.

The structure of the remaining sections in this paper is as follows: In Section 2, an overview of previous work on infrared small target detection is presented. Section 3 outlines our proposed method and details the algorithm training procedure. Experimental datasets are discussed in Section 4, where quantitative results of various methods are analyzed and the detection performance is visualized. The paper concludes with a discussion and summary in Section 5.

2. Related Work

2.1. Infrared Small Target Detection Based on a Single Frame

Frame-based infrared small target detection refers to the process of detecting small targets in individual frames of raw infrared images. Typically, this is achieved by employing image processing techniques to enhance the targets or differentiate them from the background, followed by threshold segmentation for target recognition, without considering temporal sequences. Such methods are characterized by their concise approach, low computational complexity, fast detection speed, and ease of deployment on hardware platforms.

In 2013, Gao et al. [15] proposed the Infrared Patch Image (IPI) model for infrared small target detection. The approach involves dividing the image into blocks using a sliding window, followed by vectorizing the block matrices and rearranging them into a data matrix. Robust principal component analysis (RPCA) is employed to decompose the data matrix into a low-rank matrix and a sparse matrix. Finally, the sparse matrix is reconstructed into an image, yielding the target image. In 2014, Han et al. [16] proposed a fast target acquisition method for infrared small target detection based on the human visual system (HVS) attention transfer mechanism, which incorporates thresholding operations and a fast traversal mechanism. In 2016, Wei et al. [17] proposed a multi-scale image patch contrast measurement method, inspired by biology, aiming to enhance the contrast between the target and the background, to achieve the detection of small targets through a simple adaptive threshold segmentation method. In 2016, Deng et al. [18] proposed an effective small target detection method based on weighted image entropy. This method improves the signal-to-noise ratio between interfering objects and small targets in a scene that have similar thermal intensity measurements to the background through multi-scale grayscale difference weighted local entropy measurements, combined with adaptive threshold operations. In 2016, Dai et al. [19] proposed the weighted infrared patch-image (WIPI) model, which introduced structural prior information into the separation process of weak infrared targets and background, and added an adaptive coefficient to each image block, to replace the one used in the original IPI model, which had a unified coefficient.

Traditional methods cannot cope with changing scenarios with complex backgrounds. These methods usually need to adjust the parameters of each scenario according to each scenario, which brings challenges with application in actual scenarios. In recent years, methods based on deep learning have also been applied in the field of infrared small target detection. In 2015, Cui et al. [20] first used a phase spectrum Fourier transform to extract salient areas that may contain targets, and then used SVM to classify these areas to confirm the areas of small targets. In 2017, Liu et al. [21] used a multi-layer fully connected neural network to detect small infrared targets. The neural network only classified image blocks of size 21 × 21 and identified targets and non-targets. Its limited training data limited the detection ability of the model. In 2017, Wang et al. [22] used a context aggregation network as the basic network. The networks were connected to each other using U-net form, and they proposed a loss function based on missed detection and false detection to solve the problem of the serious imbalance between target and background duty cycles. The internal attention aware network (IAANet) explores the relationship between image pixels and enhances the correlation of different targets through attention at different scales [23]. DNANet proposed the DNIM module, which improves target feature extraction and preservation effects through dense connections and spatial pyramid fusion [24].

2.2. Infrared Small Target Detection Based on Multiple Frames

The infrared small target detection method based on sequence images mainly uses the temporal and spatial information of the sequence images, combined with the continuity of the target motion and the consistency of the trajectory to detect the target.

In 2011, Qi et al. [25] proposed two improved optical flow estimation methods. By modifying the constraint function and introducing an adaptive threshold, they solved the problems of the accuracy of optical flow estimation and holes in the target area at low gradient points. In 2019, Lu et al. [26] proposed a sparse representation method based on online learning of double sparse background dictionaries for efficient detection of small infrared targets. This method achieved accurate target detection and background clutter suppression by constructing a background dictionary and using a sparse representation model to decompose the target image. In 2020, Zhao et al. [27] proposed an infrared moving small target detection algorithm that utilizes the spatiotemporal consistency of motion trajectories. By densely sampling and tracking feature points, dense trajectories are calculated, and suspicious trajectories are deleted using the motion characteristics of the target. A binary image is created based on the image coordinates of the trajectory points, extracting significant contours as candidate target areas. Through an encoding mechanism of contour numbers, the temporal consistency of contour codewords is used to distinguish moving targets and backgrounds. In 2021, Wang et al. [28] proposed a new small target detection method based on the spatiotemporal information of infrared images. By establishing a non-overlapping patch spatio-temporal tensor (NPSTT) model and introducing a tensor capped kernel norm (TCNN), potential target detection in complex backgrounds is achieved.

However, the relevant research on applying deep learning technology to multi-frame infrared small target detection is still insufficient. Kwan et al. [29] directly detected video frames using deep learning single-frame detectors, which cannot utilize temporal context information. Some methods use the spatiotemporal context information of the current frame and super-resolution operations to enhance the details of the current frame, to improve target recognition capabilities [30,31]. Yao et al. [32] used a maximum filter preprocessing operation to integrate multiple frames as input to a single-frame detection method. Sun et al. [33] used the frame to be detected as the central frame, performed a subtraction operation between the adjacent frames adapted to the central frame and the central frame to obtain a difference map, and concatenated the central frame with the two obtained difference maps before inputting it into the detection network. These methods utilize multiple frames more directly and do not take into account the redundancy of consecutive frames. At the same time, they do not fully consider the interference caused by the background motion of infrared video frames on detection. On this basis, this paper proposes an adaptive frame sampling method and a multi-frame detection network aligned at the feature map level.

3. Our Method

3.1. Adaptive Frame Sampling

In order to solve the shortcomings of uniform frame sampling and random frame sampling, this paper proposes an adaptive frame sampling method based on mutual information. The motion changes between frames are quantified by calculating the mutual information between consecutive frames, and then the motion is accumulated according to the time axis, to obtain the distribution function of the total motion change with time, and finally adaptive frame sampling is performed based on this function. The following content explains the adaptive frame sampling method in detail.

3.1.1. Mutual Information

Mutual information is a quantitative indicator that measures the correlation between two random variables. It relies on entropy and joint entropy. Entropy is a measure of the uncertainty of a random variable, and joint entropy examines the overall complexity of all possible outcomes given two random variables. Specifically, given two discrete random variables X and Y, the mutual information between them $MI(X,Y)$ can be calculated as follows:

$$\begin{array}{c}MI(X,Y)=H\left(X\right)+H\left(Y\right)-H(X,Y)\end{array}$$

(1)

where $H\left(X\right)$ and $H\left(Y\right)$ respectively are the marginal entropies of random variables X and Y, and $H(X,Y)$ is the joint entropy of X and Y. Marginal entropy and joint entropy can be calculated as follows:

$$H\left(X\right)=-\sum _{x\in X}p\left(x\right)logp\left(x\right)$$

(2)

$$H(X,Y)=-\sum _{x\in X,y\in Y}p(x,y)logp(x,y)$$

(3)

where $p\left(x\right)$ represents the probability density function of X, and $p(x,y)$ represents the joint probability density function of X and Y. If you put Equations (2) and (3) into Equation (1), you can further obtain the following equation:

$$MI(X,Y)=\sum _{x\in X,y\in Y}p(x,y)log{\displaystyle \frac{p(x,y)}{p\left(x\right)p\left(y\right)}}$$

(4)

To calculate the mutual information between two consecutive frames of infrared images, the marginal probability distribution can be obtained through from the respective histograms, and the joint probability distribution can be obtained using the joint histograms. Mutual information can measure the similarity between two frames of images [34]. The stronger the correlation between the two frames of images, the greater the mutual information. The weaker the correlation, the smaller the mutual information; that is, the more significant the motion between the current and subsequent frames. The smaller the mutual information, the more mutual information occurs when the two images are exactly the same.

Let the mutual information between the $F_{t-1}$ frame and the $F_t$ frame be denoted as $M_t$, which can be calculated as follows:

$$M_t=MI(F_{t-1},F_t)$$

(5)

where $t\in 1,2,\dots ,T$, for frame $F_0$, we simply define its mutual information $M_0$ as 0. In order to facilitate the analysis of the motion information distribution, we inverted $M_t$ and normalized it using $l_1$ norm:

$$M_t^{\prime }=\underset{i\in T}{max}\left(M_i\right)-M_t$$

(6)

$${\widehat{M}}_t={\displaystyle \frac{M_t^{\prime }}{{\sum }_T{M}_t^{\prime }}}$$

(7)

the $MI$ score ${\widehat{M}}_t$ can represent the motion information from the previous frame to the current frame. The larger the ${\widehat{M}}_t$, the more significant the motion between the two frames.

3.1.2. Adaptive Frame Sampling Based on Mutual Information

Inspired by the literature [35], we use the cumulative distribution function for frame sampling, and adaptively sample frames according to the motion information distribution of the entire set of consecutive frames to be sampled. As shown in Figure 2, given a set of consecutive frames to be sampled containing T + 1 frames, a set of $MI$ scores can be obtained by calculating the $MI$ scores of adjacent frame pairs, $S=[{\widehat{M}}_0,{\widehat{M}}_1,\dots ,{\widehat{M}}_T]$, where each element ${\widehat{M}}_t\in [0,1],t\in \{0,1,\dots ,T\}$. Then, we accumulate the obtained $MI$ scores to obtain the cumulative motion distribution function of the set of consecutive frames to be sampled:

$$M_d\left(t\right)=\sum _{i\le t}{\widehat{M}}_i$$

(8)

The obtained motion accumulation distribution function is shown in Figure 2, where the X-axis represents the frame index and the Y-axis represents the total motion accumulation up to the current frame. Finally, to sample N frames from the consecutive frames set, we divide the Y-axis evenly into N segments, and the frames whose indices fall within the segments corresponding to the X-axis are regarded as sub-sampling sets, with a total of N sub-sampling sets. Then, in order to increase the diversity of the input model data, a frame is randomly sampled from each subsampling set rather than a frame at a specific location, and in particular, for the subsampling set where the frame to be detected $F_T$ is located, the frame to be extracted is fixedly set to the frame to be detected, $F_T$. Then, they form the final sampled frame set, which is the input of the network for detection. As shown in Figure 2, this sampling method can sample more frames during periods of significant motion and fewer frames during periods of relatively static motion.

3.2. Network Architecture

In order to solve the interference caused by the motion of the infrared image background caused by the motion of the infrared detector, this paper proposes a multi-frame infrared small target detection network aligned at the feature map level. As shown in Figure 3, the proposed network consists of three parts: a feature extraction and alignment module, temporal feature fusion module, and decoding module.

3.2.1. Feature Extraction and Alignment Module

After adaptive frame sampling is performed on a set of continuous infrared images, $F=[F_0,F_1,\dots ,F_T]$ that contains $T+1$ frames, a set of infrared images, $\overline{F}=[\overline{F_1},\overline{F_2},\dots ,\overline{F_N}]$ that contains N frames is obtained as the input of the detection network, where $\overline{F_N}$ is the frame to be detected. For each frame of the input network, its features are extracted using a shared weight backbone network. This backbone network was modified from MobileNetV3 [36]. In order to better extract the detailed information of infrared small target images, the number of downsampling operations was reduced to obtain a high-resolution feature map. The detailed structure is shown in

Table 1. Network structure of backbone network.

Input Size	Operator	Exp Size	Output Channels	SE	NL	Stride
${256}^2\times 3$	Conv2d,3	-	16	-	HS	1
${256}^2\times 16$	Bneck,3	16	16	True	RE	2
${128}^2\times 16$	Bneck,3	72	24	False	RE	1
${128}^2\times 24$	Bneck,3	88	24	False	RE	1
${128}^2\times 24$	Bneck,5	96	40	True	HS	2
${64}^2\times 40$	Bneck,5	240	40	True	HS	1
${64}^2\times 40$	Bneck,5	240	40	True	HS	1
${64}^2\times 40$	CCA	-	40	-	-	-

. Except for the first row, each row of operation operators is connected in series to form the backbone network. Input in the first row represents the resolution and number of channels of the input feature map of the operation operator in that row, and Operator represents the operation operator, among which Conv2d,3 represents a two-dimensional convolution with a convolution kernel of 3 concatenated with a standard convolution block of batch normalization and an activation function. Bneck represents the basic bottleneck structure, using depthwise separable convolution [37]. CCA stands for cross attention module [38], which serves as the last layer of the backbone network, to strengthen the local important information of the spatial features obtained. Exp Size represents the number of channels of the feature map in Bneck after the first Dwconv. Out Channel represents the number of channels of the feature map after this operator. SE represents the channel attention network [39], True indicates that the Bneck uses channel attention, and False indicates that channel attention is not used. NL represents what activation function the operator uses, where HS represents the HardSwish activation function and RE represents the ReLU activation function. Stride represents the step size of the operator. If the step size is 1, the resolution of the input and output feature maps will remain unchanged. If the step size is 2, the feature map will be downsampled.

The feature alignment module is shown in Figure 4. Inspired by the literature [40], the feature map to be aligned $f_i$ and the current feature map $f_N$ are concatenated and fed into the homography estimation network to obtain the corresponding homography flow matrix. Then, a warp operation is performed on the $f_i$ to obtain an aligned feature map. The aligned feature map ${f_i}^{\prime }$ is subtracted from the feature map $f_i$ to obtain a differential feature map $f_s$, and is concatenated with ${f_i}^{\prime }$ as the output $f_{out}$. In order to ensure the quality of alignment, we also introduce a triplet loss for the feature alignment module, which is explained in detail in Section 3.3. As shown in Figure 4, the estimator in the feature alignment module consists of the last three layers of ResNet18 [41], and finally eight weights are obtained through two fully connected layers.

3.2.2. Temporal Feature Fusion Module

The spatiotemporal fusion module is modified from 3D ResNets [42]. Its structure is shown in Figure 5. It consists of three large residual units connected in series. The first residual unit has a step size of 1 and does not downsample the feature map in the time channel, while the last two residual units have a stride of 2, and the feature map is downsampled four times in the time channel in total. Each residual unit comprises four basic blocks connected in series. The first basic block has a different step size, and the remaining basic blocks have a step size of 1. Each basic block is composed of a 3D convolution structure superimposed with a time-adaptive module (TAM) [43]. In the basic block, conv3d, k = 3, s = S represents a 3D convolution with a kernel size of 3 and a stride of S. A standard 3D convolutional combination of batch normalization layers in series with ReLU activation functions is used. The TAM module enhances the network’s ability to capture important temporal information by learning adaptive kernels.

3.2.3. Decoding Module

The decoding network structure is shown in Figure 6. The input feature map first passes through two standard deconvolution layers with a step size of 2. Each standard deconvolution layer is composed of a two-dimensional deconvolution series batch normalization with a convolution kernel of 4. There is a unified layer and ReLU activation function, then the resolution of the feature map is restored to the same size as the input image resolution. Then the feature map goes through a standard convolution operation layer and a two-dimensional convolution to reduce the number of channels to 1, and finally passes through the sigmoid function to output a heat map with a value between 0 and 1. By decoding the heat map, the position coordinates of the small infrared target can be obtained.

Most infrared small target detection methods use datasets with segmentation binary masks as real labels. However, when the size of the target is too small, it is difficult to distinguish clear edges. At this time, it is very difficult to use segmentation binary masks for labeling. Only the center point can be used to mark the target. However, if only a binary mask is used to mark the center, and only the center point is 1 and the rest is 0, this will cause a serious problem of imbalance between positive and negative samples. In order to solve this problem, we use the two-dimensional Gaussian distribution to create the real labels of small infrared targets. Specifically, the real label can be constructed using the following formula:

$$g_t(m,n)=exp\left(-{\displaystyle \frac{1}{2{\delta }^2}}\left[{\left(m-t_x^i\right)}^2+{\left(n-t_y^i\right)}^2\right]\right)$$

(9)

where $\delta$ is the standard deviation, i is the index label of the independent target, and $t_x^i$ and $t_y^i$ represent the x-coordinate and y-coordinate of the i-th target point. Using the two-dimensional Gaussian distribution to construct the ground truth label expands the non-zero part of the heatmap.

3.3. Loss Function

In the feature alignment module, in order to ensure the quality of alignment and guide the learning direction of the alignment module, we introduce a triplet loss without involving an attention mask as in [44], i.e.,

$$L_t=\sum _{i=1}^{N-1}|f_i^{\prime }-f_N|-|f_i-f_N|$$

(10)

where $f_i,f_i^{\prime }$ are the feature map before and after alignment, and $f_N$ is the current feature map.

For infrared small targets, the area produced by the two-dimensional Gaussian expansion of the true label is still insufficient. In order to further balance the positive and negative samples, a mask loss is introduced to supervise the learning of the model, The mask matrix M is defined as follows:

$$M(m,n)=\left\{\begin{array}{cc}\hfill 1,\hfill & \hfill g_t(m,n)>\alpha \hfill \\ \hfill 0,\hfill & \hfill \mathrm{else}\hfill \end{array}\right.$$

(11)

where $\alpha$ is the probability threshold, and finally our loss function can be expressed by the following formula:

$$L_h={\parallel p_t-g_t\parallel }_2\odot (M+\beta (1-M))$$

(12)

where ⊙ means element-wise multiplication, and $\beta$ in the formula is used to control the proportion of background areas in supervision. Since the number of negative samples of small infrared targets is much larger than the positive samples, the value of $\beta$ will be set smaller to balance the positive and negative samples. The value of $\alpha$ should be set to a number slightly larger than 0 but not too large, to ensure that M accurately represents the real target area. In this paper, after some experiments, $\alpha$ was set to 0.2 and $\beta$ was set to 0.3. The final total loss function can be written as

$$L=\lambda L_t+L_h$$

(13)

Since the value of $L_t$ is much larger than $L_h$, in order to ensure the dominance of $L_h$, we set $\lambda$ to 0.01 to highlight the dominance of $L_h$.

4. Experiments

4.1. Datasets

This paper used the real low-altitude scene infrared small target dataset [45] published by the National University of Defense Technology for training and testing the network. The typical wavelength range of this dataset is 3–5 µm, which belongs to the medium-wave infrared. There are 22 typical scenes, which mainly include a single target with a low-altitude complex background, targets moving from far to near, targets moving from near to far, the target leaving the field of view, the target returning to the field of view, etc. Most of the scenes in the dataset are ground backgrounds, including forests, rivers, buildings, etc. Each typical scene corresponds to a data sequence, with a total of 22 data sequences, 16,177 image frames, and 16,944 targets with labels. Each infrared image has a resolution of 256 × 256 pixels and a bit depth of 8 bits. Typical small target samples from the dataset are shown in Figure 7. The environmental background of the dataset and the signal-to-clutter ratio (SCR) information of the data are shown in

Table 2. Scenarios of each data segment.

Data Segment	Frames	Average SCR	SCR Variance
Data1	399	9.72	0.033
Data2	599	4.34	0.220
Data3	100	2.17	0.908
Data4	399	3.75	3.646
Data5	3000	5.45	1.285
Data6	399	5.11	1.571
Data7	399	6.33	20.316
Data8	399	6.07	0.159
Data9	399	6.29	17.086
Data10	401	0.38	0.031
Data11	745	2.88	2.148
Data12	1500	5.20	2.226
Data13	763	1.98	0.886
Data14	1426	1.51	1.538
Data15	751	3.42	0.965
Data16	499	2.98	0.674
Data17	500	1.09	0.353
Data18	500	3.32	0.165
Data19	1599	3.84	0.886
Data20	400	3.01	1.485
Data21	500	0.42	0.092
Data22	500	2.20	0.150

. The definition of SCR is as follows:

$$SCR={\displaystyle \frac{\left|{\gamma }_{tg-}{\gamma }_{img}\right|}{{\sigma }_{img}}}$$

(14)

where ${\gamma }_{tg}$ is the brightness of the target area, ${\gamma }_{img}$ is the average brightness of the image, and ${\sigma }_{img}$ is the standard deviation of the image brightness.

4.2. Evaluation Metrics

This paper used five evaluation indicators [17,46] to evaluate the proposed method, including $precision$, $recall$, $F_1$ score, detection rate $P_d$, and false alarm rate $F_a$. They are defined as follows:

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(15)

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(16)

$$F_1={\displaystyle \frac{2·Precision·Recall}{Precision+Recall}}$$

(17)

$$P_d=Recall$$

(18)

$$F_a=1-Precision$$

(19)

In the above formulas, $TP$ represents the correct prediction, specifically referring to the number of predicted points within d pixels of the distance between the predicted point and the actual point. $FP$ represents an incorrect prediction, specifically referring to the number of predicted points that do not have actual points within d pixels around the predicted point. $FN$ represents the actual points that were missed, specifically referring to the number of actual points that have no predicted points within d pixels around the actual point. The value of d in this paper was set to 10. The $precision$ represents the ratio of the number of correctly predicted targets to the total number of predicted targets, the $recall$ rate represents the ratio of the number of correctly predicted targets to the total number of real targets, and the $F_1$ score can comprehensively reflect the detection performance of the model.

4.3. Implementation Details

The method in this article was implemented using the PyTorch 1.10 deep learning framework. The computer platform was equipped with an Intel i7-6700 CPU and a NVIDIA GeForce GTX1080Ti GPU. The model was trained under the Ubuntu 20.04.2 LTS operating system environment. The settings during training included total training epochs of 100, using the Adam optimizer, where the batch size was 8. A dynamic learning rate strategy was sampled during training, with an initial learning rate of 0.0005. The learning rate was gradually reduced according to the cosine law within 66 epochs, and the final learning rate was 0.00025.

4.4. Experimental Results

4.4.1. Comparisons with Other Methods

In order to directly compare with the original results of the comparison methods, according to the different performance indicators, the method in this paper was tested on three groups and compared qualitatively and quantitatively with methods in other literature works. Data5, Data6, Data15, Data19, Data20, and Data22 in group 1 were used as the test set, and the rest were used as the training set. Data8 in group 2 was used as the test set, and Data2 and Data8 in group 3 were used as the test set. In group 1, the method in this paper was related to multi-scale patch-based contrast measurement (MPCM) [17], multi-scale relative local contrast measurement (RLCM) [46], spatiotemporal local contrast filter (STLCF) [18], and maximum filter, and compared with the four benchmark methods of median filter time fusion (MMTF) [47]. The performance indicators tested were detection rate and false alarm rate. The results are shown in

Table 3. Test results of different methods for each data segment in group 1.

Method	Data5		Data6		Data15		Data19		Data20		Data22
	${\mathit{P}}_{\mathit{d}}$ (%)	${\mathit{F}}_{\mathit{a}}$ (%)	${\mathit{P}}_{\mathit{d}}$ (%)	${\mathit{F}}_{\mathit{a}}$ (%)	${\mathit{P}}_{\mathit{d}}$ (%)	${\mathit{F}}_{\mathit{a}}$ (%)	${\mathit{P}}_{\mathit{d}}$ (%)	${\mathit{F}}_{\mathit{a}}$ (%)	${\mathit{P}}_{\mathit{d}}$ (%)	${\mathit{F}}_{\mathit{a}}$ (%)	${\mathit{P}}_{\mathit{d}}$ (%)	${\mathit{F}}_{\mathit{a}}$ (%)
MPCM	68.58	47.99	62.25	52.35	35.05	31.96	84.16	25.33	76.11	38.28	39.54	34.49
RLCM	69.56	4.77	45.40	4.88	39.50	4.54	81.39	4.12	81.90	4.13	40.39	5.13
STLCF	65.74	3.92	73.59	3.94	57.39	3.95	60.83	3.85	67.24	3.87	82.70	3.91
MMTF	33.26	4.19	56.15	4.02	47.84	4.09	24.73	3.95	40.12	3.93	35.99	3.98
Ours	99.8	0.33	100	0	99.6	4.72	97.94	2.2	95.5	4.97	96.6	0

. The bolded parts in the table indicate the best effect of rearrangement. It can be seen that, in terms of detection rate and false alarm rate, the detection performance of this method on six data segments was greatly improved compared to the three benchmark methods of MPCM, RLCM, and MMTF. Although the false alarm rate on data segment 20 was slightly higher than the benchmark method STLCF, the detection rate of this method was higher than STLCF, and the comprehensive detection performance was also better than STLCF.

In group 2, the method in this paper was compared with another method from the literature [48]. The performance indicators of the test were detection rate and false alarm rate. The results are shown in

Table 4. Test results of different methods for each data segment in group 2.

Method	Data8
	${\mathit{P}}_{\mathit{d}}$ (%)	${\mathit{F}}_{\mathit{a}}$ (%)	${\mathit{F}}_{\mathbf{1}}$ (%)
Yan et al. [48]	98.12	14.35	91.46
Ours	97.80	1.2	98.30

. The method proposed in this article was slightly better than the method from the literature [48]. At the same time, the false alarm rate of this method on data segment 8 was reduced by 13% compared with the other method, and the comprehensive detection performance $F_1$ score was 6.84% better than the other method.

In group 3, the method in this paper was compared with the method in [32]. The performance indicators of the test were accuracy, recall rate, and score. The results are shown in

Table 5. Test results of different methods for each data segment in group 3.

Method	Data2			Data8
	$\mathit{Precision}$ (%)	$\mathit{Recall}$ (%)	${\mathit{F}}_{\mathbf{1}}$ (%)	$\mathit{Precision}$ (%)	$\mathit{Recall}$ (%)	${\mathit{F}}_{\mathbf{1}}$ (%)
Yao et al. [32]	98.9	99.6	99.2	98.4	99.2	98.8
Ours	99.3	99.5	99.4	100	99.0	99.5

. On data segment 2 and data segment 8, the accuracy of this method and the $Precision$, $Recall$, and $F_1$ score were all better than the method in [32].

The method in this paper was better than the traditional method in terms of real-time performance. A comparison of the inference speed with each method can be seen in

Table 6. Inference time and fps between our model and other methods.

Method	Input Size	Inference Time (s)	FPS
MPCM	[256,256]	0.5208	1.92
RLCM	[256,256]	4.1667	0.24
STLCF	[256,256]	1.3513	0.74
MMTF	[256,256]	0.6329	1.58
Yan et al. [48]	[256,256]	0.0596	16.78
Yao et al. [32]	[256,256]	0.0282	35.50
Ours	[256,256]	0.0269	37.14

. When the resolution of the input frame was $256\times 256$, the network inference time of this method was 0.0269, which can meet real-time requirements.

4.4.2. Ablation Experiment

We conducted ablation experiments to investigate the impact of the actual number of frames n input into the network and the total number of consecutive frames N used for frame sampling on the detection results. We obtained N by multiplying n by a positive integer $\mu$, and the experiments were conducted on group 1. The impact of different actual input frame numbers n on detection performance is shown in the

Table 7. Quantitative results of ablation experiments with number of network input frames.

n	${\mathit{F}}_1$ (%)	Flops (G)	Parameters (M)	Inference Time (s)
4	98.6	43.17	4.62	0.0269
8	98.9	84.32	6.78	0.0423
16	99.0	178.73	9.43	0.1153

. According to Figure 8, the $F_1$ score of the model was the highest when $\mu$ was 5. Therefore the results in the table were all experimentally obtained under the condition that $\mu$ was 5. As n increased, the score of the model also increased slightly, but the Flops, model Parameters, and Inference Time of the model increased sharply, and the corresponding index score reflecting the accuracy of model detection only increased to a limited extent. Considering these factors comprehensively, this paper set the actual number of frames input to the network to 4.

The influence of $\mu$ on detection performance is shown in Figure 8. In the experiment, n was set to 4. It can be observed that the detection performance of the model increased with the increase in $\mu$ within a certain range. However, when $\mu$ exceeded 5, increasing $\mu$ not only failed to improve the detection performance but could even lead to a decrease. This is because introducing more frames can enrich the target’s motion information. However, when the time span of the introduced frames is too large, the continuity between the introduced frames and the frames to be detected becomes smaller, causing interference in the detection process.

In order to verify the effectiveness of the proposed adaptive frame sampling method and feature alignment detection network, this paper designed a series of ablation experiments. All ablation experiments were performed on group 1, n was set to 4 and $\mu$ was set to 5.

For the proposed frame sampling method, we conducted experiments on no sampling, random sampling, uniform sampling, and adaptive frame sampling, respectively. Sampling effect diagrams of the various sampling methods are shown in Figure 9, and the results are shown in

Table 8. Quantitative results of ablation experiments with different sampling methods.

Method	Precision (%)	Recall (%)	${\mathit{F}}_1$ (%)	${\mathit{P}}_{\mathit{d}}$ (%)	${\mathit{F}}_{\mathit{a}}$ (%)
Without Sampling	95.6	96.04	95.82	96.04	4.39
Random Sampling	97.3	97.5	97.39	97.5	2.72
Uniform Sampling	97.4	97.7	97.5	97.7	2.61
Adaptive Sampling	98.5	98.74	98.61	98.7	1.52

. Under the same computational cost, using frame sampling can introduce more motion information, thereby enhancing the detection performance. At the same time, compared with fixed strategy sampling methods such as random sampling and uniform sampling, the adaptive frame sampling method proposed in this paper could sample more reasonably, making the sampled frame motion distribution more uniform.

For the proposed feature alignment module, in order to verify its effectiveness, ablation experiments were conducted on each component to verify its effectiveness. The experimental results are shown in

Table 9. Quantitative results of ablation experiments with components of FAM.

FAM	Feature Minus	Triplet Loss	Precision (%)	Recall (%)	${\mathit{F}}_1$ (%)	${\mathit{P}}_{\mathit{d}}$ (%)	${\mathit{F}}_{\mathit{a}}$ (%)
✗	✗	✗	93.9	94.7	94.3	94.7	6.06
✔	✗	✗	97.0	96.3	96.6	96.3	2.98
✔	✔	✗	97.1	97.9	97.5	97.9	2.87
✔	✗	✔	97.9	96.4	97.1	96.4	2.09
✔	✔	✔	98.5	98.7	98.6	98.7	1.52

. ✔ in the table indicates retaining the module in that column, and ✗ indicates removing the module. FAM (feature alignment module) is feature alignment module, which is explained in Section 3.2.1, and the experimental results show that each component in the proposed feature alignment module had a positive impact on the model performance.

5. Discussion

In this paper, we proposed an adaptive frame sampling method based on mutual information, to expand the input motion information of a multi-frame infrared small target detection model without increasing the computational complexity. At the same time, a multi-frame based infrared small target detection network was proposed that is aligned at the feature map level, and the feature maps are aligned by estimating the homography flow matrices. Experimental results demonstrated the effectiveness of our proposed method. Our proposed adaptive frame sampling method could sample more reasonably and make the sampled frame motion distribution more uniform. The proposed detection network containing a feature alignment module can be trained end-to-end, which enhances the robustness of the model. Our detection model could achieve an inference speed of 37 fps with a detection rate of 98% and a false alarm rate of 1% and a $F_1$ score of 98.5% on a NVIDIA 1080Ti GPU with 4 frames of input.