Temporal Denoising of Infrared Images via Total Variation and Low-Rank Bidirectional Twisted Tensor Decomposition

Abstract

Temporal random noise (TRN) in uncooled infrared detectors significantly degrades image quality. Existing denoising techniques primarily address fixed-pattern noise (FPN) and do not effectively mitigate TRN. Therefore, a novel TRN denoising approach based on total variation regularization and low-rank tensor decomposition is proposed. This method effectively suppresses temporal noise by introducing twisted tensors in both horizontal and vertical directions while preserving spatial information in diverse orientations to protect image details and textures. Additionally, the Laplacian operator-based bidirectional twisted tensor truncated nuclear norm (bt-LPTNN), is proposed, which is a norm that automatically assigns weights to different singular values based on their importance. Furthermore, a weighted spatiotemporal total variation regularization method for nonconvex tensor approximation is employed to preserve scene details. To recover spatial domain information lost during tensor estimation, robust principal component analysis is employed, and spatial information is extracted from the noise tensor. The proposed model, bt-LPTVTD, is solved using an augmented Lagrange multiplier algorithm, which outperforms several state-of-the-art algorithms. Compared to some of the latest algorithms, bt-LPTVTD demonstrates improvements across all evaluation metrics. Extensive experiments conducted using complex scenes underscore the strong adaptability and robustness of our algorithm.

1. Introduction

Infrared focal plane arrays (IRFPAs) are extensively used in the defense, medical, industrial, and security fields [1,2,3]. Specifically, they play a crucial role in applications such as target detection and tracking (especially in remote sensing scenarios where atmospheric turbulence and sensor limitations introduce significant noise), situational awareness, non-destructive testing, and medical diagnosis [4,5,6]. Moreover, IRFPAs are vital for surveillance under low-light or adverse weather conditions, where noise levels are often higher, impacting the visibility of small targets [7]. Point target detection in IRFPA imaging faces several challenges. Key among these are two linearly independent interferences, random unstructured noise and structured background clutter. While noise appears as random fluctuations, clutter manifests as structured patterns from complex background scenes that may be mistaken for targets. In addition to these, the inherent characteristics of IRFPA detectors introduce further complexities. The response of each unit in the detector array is inconsistent and unstable [8,9]. This is due to factors such as the detector material characteristics, manufacturing processes, and optical systems. This inconsistency leads to the superimposition of spatial fixed-pattern noise (FPN) and temporal random noise (TRN) on the scene image. FPN manifests as a consistent spatial pattern, while TRN varies randomly over time. Background clutter, on the other hand, arises from the complexity and variability of the scene itself. This issue is particularly prominent for uncooled IRFPA, which has low sensitivity and is more susceptible to FPN and TRN, significantly affecting the imaging quality [10]. This is especially relevant for surveillance applications using low-cost, uncooled infrared detectors, which are generally more susceptible to noise than cooled ones [11,12]. In practical applications, TRN can significantly degrade the performance of IRFPA systems. For instance, it can reduce the signal-to-noise ratio and contrast of the image, mask the detailed information of the target, and introduce false alarms and missed detections, thereby reducing the reliability of target detection and tracking. In practical applications, TRN can markedly diminish the performance of IRFPA systems by reducing the image’s signal-to-noise ratio and contrast, obscuring detailed target information, and leading to false alarms and missed detections. This paper focuses on addressing the challenge of TRN suppression in infrared small target detection, recognizing its significant impact on system performance.

Image denoising is an important step in object recognition tasks. It offers several benefits, including improved accuracy and recall rate, enhanced adaptability to complex environments, and increased algorithm robustness [13,14,15]. This ultimately leads to more stable and reliable detection systems. Furthermore, in the context of infrared imaging, effective denoising is critical for enhancing the performance of various applications. For example, in target detection and tracking, denoising can improve the detection range and accuracy, especially for small or low-contrast targets. In addition, our proposed denoising method can serve as a preprocessing step to improve the input and enhance the performance of existing target tracking algorithms [16,17,18]. In situational awareness, denoising can enhance the clarity and interpretability of infrared images, leading to better understanding of the environment. In non-destructive testing and medical diagnosis, denoising can improve the sensitivity and reliability of defect or disease detection. The denoising process can be performed separately from the object recognition task or completed simultaneously as a whole. For deep learning-based object recognition methods, networks with deeper layers have good feature extraction capabilities, and specifically designing a denoising step may lead to limited improvement in recognition accuracy and affect computational efficiency [19]. For most object recognition methods, the impact of dataset quality is more obvious, and denoising preprocessing can effectively improve recognition accuracy [20,21,22,23].

FPN generally manifests as fixed patterns in the spatial domain, and relatively good correction effects can be achieved through methods such as calibration or scene-based approaches [24,25,26,27]. While background clutter can be mitigated through sophisticated background estimation and subtraction techniques, the TRN in IRFPAs predominantly appears as horizontal and vertical stripe noise [10]. Although often approached as spatial noise, standard IRFPA calibration techniques for stripe noise, be it FPN or TRN, typically depend on single-frame data. This approach neglects the potential benefits of exploiting temporal correlation. While the significance of temporal noise has been acknowledged [28], subsequent research [29] underscores the need for developing calibration methods that effectively leverage inter-frame relationships. While research on TRN suppression in infrared imagery is still developing and offers a limited number of dedicated methods, the field of video restoration boasts a wide array of established techniques, including temporal filtering, motion estimation, and motion compensation [30]. These methods, commonly employed in visible video denoising, leverage the temporal correlation in video sequences. This reliance on inter-frame relationships, prevalent in both video restoration and TRN reduction, supports the perspective that the denoising process for TRN can be fundamentally framed as a video restoration problem [31].

Deep learning-based methods are commonly used for visible video denoising [31,32,33,34,35,36,37], which also has reference significance for the infrared band. Although significant progress has been made using these methods, they generally involve the synchronized processing of multiple frames to maintain inter-frame correlation, resulting in high memory usage. The overall hardware costs for model training, deployment, and other aspects of noise reduction remain high [32,38]. Traditional video denoising methods are often developed from the field of image denoising [39,40], with some ability to resist detail blurring. However, there are issues to varying degrees, including blurred details, poor temporal consistency, extremely high computational complexity, and limited denoising effectiveness [39,41,42,43]. The denoising process utilizing tensor recovery methods is framed as a matrix or tensor recovery problem [44,45]. This approach employs techniques such as total variation and low-rank tensor decomposition for solving. Given that the inter-frame correlation is significant, this method shows great potential and has garnered considerable attention in recent years. However, the effectiveness of tensor recovery methods in video denoising, particularly regarding denoising performance and detail preservation, remains limited [46,47]. Additionally, when these tensor recovery methods are applied for TRN suppression in hyperspectral denoising, noticeable detail blurring occurs in moving targets [48,49]. Furthermore, the effectiveness of tensor recovery methods for TRN suppression in infrared small-target recognition is relatively weak [50,51].

Infrared target tracking heavily relies on temporal information between consecutive frames to accurately estimate the target’s motion trajectory [52,53]. However, traditional single-frame denoising methods ignore this crucial temporal correlation, limiting their performance. To better exploit this spatiotemporal information, we construct the image sequence into a three-dimensional tensor and propose a tensor decomposition-based denoising method. This paper proposes a TRN correction algorithm, named bidirectional twisted Laplacian and total-variation-regularized tensor decomposition (bt-LPTVTD), based on total variation and low-rank tensor decomposition, aiming to effectively correct infrared images affected by TRN. This method fully considers the correlation of temporal and spatial information. By employing advanced tensor decomposition techniques, the algorithm can both suppress noise and preserve crucial target motion information.

The remainder of this paper is organized as follows. Section 2 provides a brief overview of the representative tensor-decomposition-based image processing and denoising methods. Section 3 presents the notation and preliminary information related to tensor decomposition used in this paper. Section 4 describes the model and algorithm for the proposed bt-LPTVTD method. Section 5 presents the experimental analysis and results used to validate the proposed method. Finally, Section 6 highlights the major conclusions drawn from the findings of this study and directions for future research.

2. Related Works

Temporal denoising necessitates expanding the research focus from a two-dimensional matrix to a three-dimensional tensor to effectively leverage inter-frame correlation for improved denoising outcomes. TRN suppression methods can be classified into three categories: deep learning methods, traditional video denoising methods, and tensor recovery methods. The primary focus of tensor recovery methods is on developing tensor decomposition techniques and constructing tensor recovery models. Significant research has been conducted in the fields of hyperspectral image denoising and infrared small-target recognition in these areas [54], providing valuable insights for TRN suppression.

2.1. Deep Learning Methods

Deep learning-based temporal noise suppression methods are frequently applied to visible video denoising and can mainly be divided into two categories: convolutional neural network (CNN)-based methods and transformer-based methods. These models typically incorporate motion estimation or video frame alignment modules to strike a balance between effective denoising and the preservation of motion details.

In 2017, Zhang et al. [33] introduced a blind Gaussian denoising method utilizing a single deep convolutional neural network (CNN) for handling unknown noise levels. This method used residual learning to separate noise and included batch normalization to speed up the training process and improve denoising performance. In 2019, Claus et al. [55] developed Deep Blind Video Denoising (ViDeNN), which integrates spatial and temporal filtering within a CNN framework to achieve blind denoising while ensuring a degree of temporal consistency. Tassano et al. followed this with the Deep Video Denoising network (DVDnet) [34] and FastDVDnet [32] in 2019 and 2020, respectively. These approaches separate video noise into spatial and temporal components and employ two modules trained in two stages, resulting in enhanced denoising performance and significantly reduced computation time. In 2021, Sheth et al. [35] proposed an unsupervised deep video denoising (UDVD) method that trains exclusively on noisy data, achieving results comparable to those of supervised methods. In 2022, Monakhova et al. [56] leveraged generative adversarial networks (GANs) for video denoising in extremely low-light conditions. Additionally, Song et al. [37] introduced TempFormer, a hybrid model that combines temporal and spatial transformer blocks with three-dimensional convolutional layers to deliver high-quality video denoising. In 2024, Liang et al. [31] presented the Video Restoration Transformer (VRT) model, designed for tasks such as super-resolution, deblurring, denoising, frame interpolation, and spatiotemporal video super-resolution.

In recent years, deep learning-based video denoising methods have garnered significant attention and achieved remarkable advancements in denoising performance. However, these approaches have considerable memory requirements when processing image data, and video denoising methods that account for inter-frame correlations typically process multiple frames simultaneously, further increasing memory usage. As a result, the overall hardware costs for training, deployment, and related processes remain high. Moreover, deep learning methods often struggle to train effectively on real noisy images, and most denoising techniques are restricted to specific types of noise, limiting their applicability [57].

2.2. Traditional Video Denoising Methods

Traditional video denoising methods typically derive from image denoising techniques and frequently employ block-matching methods.

The traditional method for video denoising based on block matching is known as block-matching and 4D filtering (BM4D). This technique evolved from block-matching and 3D filtering (BM3D) and was introduced by Maggioni et al. [39,40] in 2012. Initially, BM4D was applied to denoise magnetic resonance images (MRI), but has since become commonly applied in video denoising. In 2014, Sutour et al. [41] presented a method for image and video denoising based on total variation, utilizing 3D blocks for spatiotemporal denoising. This approach integrated non-local means with total variation (TV) regularization, addressing the issues of high computational complexity and sensitivity to noise that are inherent in block-matching techniques. In 2018, Arias et al. [42] introduced a video denoising method grounded in empirical Bayesian estimation, which constructs a Bayesian model for each group of similar spatiotemporal blocks, thus mitigating issues arising from motion estimation inaccuracies. In 2019, Li et al. [43] achieved image and video recovery using a multi-plane autoregressive model combined with low-rank optimization.

In addition to block-matching methods, there are other approaches to video denoising that often incorporate block-matching techniques. In 2012, Reeja et al. [58] achieved video denoising by separating the background from the moving parts, applying temporal averaging and spatial filtering independently, which resulted in superior denoising performance compared to K-nearest filtering, while also fulfilling real-time processing requirements. In 2015, Han et al. [59] developed a nonlinear estimator for video denoising in the frequency domain, which simultaneously conducted motion estimation and noise observation, allowing for integration into video compression codecs to enhance computational efficiency. In the same year, Ponomaryov et al. [60] introduced a three-stage framework for denoising color video sequences affected by additive Gaussian noise, comprising spatial filtering, spatiotemporal filtering, and spatial post-processing. In 2020, Samantaray et al. [61] proposed a rapid trilateral filter for video denoising.

The 3D video denoising approaches of these traditional methods resemble those used for 2D images, as they consider temporal correlation through motion estimation techniques such as block matching. These approaches achieve video denoising with some resistance to detail blurring. However, due to the limited or costly utilization of inter-frame correlation, these methods often encounter issues such as detail blurring, poor temporal consistency, extremely high computational complexity, and limited effectiveness in denoising.

2.3. Tensor Recovery Methods

Tensor recovery methods consider image sequence blocks as the objects of processing and express them as matrix or tensor recovery problems, employing techniques such as total variation and low-rank tensor decomposition for their resolution. These methods effectively suppress temporal noise by operating jointly in the spatiotemporal domain, taking into account significant inter-frame correlations. As a result, they possess considerable potential and have garnered widespread attention in recent years.

Tensor recovery methods are applied in video restoration, where tensors consist of multiple temporally continuous images. These methods typically decompose the tensor into a recovery component and a noise component, with noise and interference characterized by the Frobenius norm, primarily focusing on recovery techniques. In 2017, Gui et al. [47] represented video 3D spatiotemporal blocks as a tensor structure and utilized low-rank tensor decomposition for denoising. This approach initially demonstrated the feasibility of tensor decomposition in video denoising. However, the simplicity of the tensor decomposition process makes it challenging to balance denoising and blur suppression. In the same year, Hu et al. [62] introduced a twisted tensor to optimize the tensor singular value decomposition (t-SVD) process by leveraging the temporal correlation of the video. They proposed the twisted tensor nuclear norm (t-TNN), which effectively preserves the details of moving objects. In 2022, Qiu et al. [63] introduced the tensor ring nuclear norm for estimating the recovery tensor, significantly enhancing computational efficiency. Additionally, Shen et al. [46] employed low-rank tensor decomposition to extract the background for foreground and background segmentation of noisy videos, using total variation to extract the foreground.

Beyond video restoration, tensor recovery methods have also found applications in hyperspectral image denoising and infrared small-target detection. In hyperspectral image denoising, methods like total variation regularized low-rank tensor decomposition [49], weighted norms combined with low-rank tensor decomposition [64], and variations of tensor nuclear norm [48,65,66] have been proposed to exploit spatial–spectral correlations and address mixed noise. For infrared small-target detection, tensor-based approaches often decompose the data into background, target, and noise components, with methods like asymmetric spatial–temporal total variation (ASTTV) regularization [51,67], truncated nuclear norm [68], and twisted tensor nuclear norm [50] being used to separate and recover the target of interest. However, many tensor-based video denoising methods are relatively simplistic, leading to limited effectiveness in noise reduction and detail preservation. For instance, while powerful in their domain, hyperspectral image denoising methods like LRTDTV [49] often struggle with temporal noise in videos, leading to blurred moving targets as they treat data as static sequences (Figure 1c). Similarly, infrared small-target recognition methods, such as ASTTV-NTLA [51], prioritize preserving dynamic targets but may compromise noise suppression, leading to a trade-off between denoising strength and detail preservation (Figure 1d,e). Therefore, achieving optimal results often requires careful parameter tuning, and further research is needed to develop more sophisticated tensor-based methods that can effectively address the challenges of temporal denoising in infrared image sequences.

3. Notations and Preliminaries

3.1. Notations

To ensure clarity in the subsequent sections, a detailed explanation of the symbols and theorems utilized in this study is provided beforehand. The symbols used in this study are listed in

Table 1. Notations.

Notations	Explanations
$\mathcal{X}/X/\mathit{x}/x$	Tensor/Matrix/Vector/Scalar
${\mathcal{X}}_{i,j,k}$	The (i, j, k)-th element
${\mathcal{X}}_{:,:,k}$ or ${\mathcal{X}}^{(k)}/{\mathcal{X}}_{:,j,:}/{\mathcal{X}}_{i,:,:}$	Temporal/Vertical/Horizontal slice
${\mathcal{X}}^i$	The i-th iteration value of $\mathcal{X}$
${\mathcal{X}}_{(i)}\in {\mathbb{R}}^{n_i\times {\prod }_{k\ne i}{n}_k}$	Mode-i decomposition of tensor $\mathcal{X}$
$⟨\mathcal{X},\mathcal{Y}⟩$	Inner product of tensors $\mathcal{X}$ and $\mathcal{Y}$
${\Vert \mathcal{X}\Vert }_F$	Frobenius norm of tensor $\mathcal{X}$
$\Vert \mathcal{X}{\Vert }_{*}$	Nuclear norm of tensor $\mathcal{X}$, the sum of all singular values
$\mathcal{Y}=\mathcal{X}{\times }_{n}U$	The Mode-n product of $\mathcal{X}$ and $U$, represented in matrix form as $Y_{(n)}=U{X}_{(n)}$
$\overrightarrow{\mathcal{X}}$	The twisted tensor of $\mathcal{X}$

.

3.2. Anisotropic Spatiotemporal Total Variation Regularization

Because TV regularization preserves the spatial smoothness, edge structure, and spatial sparsity of images, it is widely used in image restoration, small-target recognition, and other fields [49,69,70,71,72]. However, most existing total variation denoising methods process only spatial 2D images, which can only describe the spatial characteristics of scene information and ignore the temporal changes in the target. If each image is separately subjected to spatial total variation denoising, it not only takes time but also ignores the smoothness characteristics of the scene in the temporal domain. Therefore, a more efficient total variation denoising method that can utilize the continuity of the target scene in both the spatial and temporal domains is required.

Owing to the strong correlation of scene information between consecutive frames and the smoothness exhibited in local spatial regions, asymmetric spatiotemporal total variation (ASTTV) can be used to simultaneously model the continuity of both space and time. ASTTV extends the 2D spatial TV to the 3D spatiotemporal domain, considering both the temporal and spatial total variations and combining them into a regularization term:

$${\Vert \mathcal{X}\Vert }_{ASTTV}={\Vert D_h\mathcal{X}\Vert }_1+{\Vert D_v\mathcal{X}\Vert }_1+\delta {\Vert D_t\mathcal{X}\Vert }_1$$

(1)

where D_h, D_v, and D_t represent differential operators in the horizontal, vertical, and temporal dimensions, respectively. δ is a constant that controls the weighting of temporal differences. D_h, D_v, and D_t are defined as follows:

$$D_h\mathcal{X}(i,j,k)=\mathcal{X}(i+1,j,k)-\mathcal{X}(i,j,k)$$

(2)

$$D_v\mathcal{X}(i,j,k)=\mathcal{X}(i,j+1,k)-\mathcal{X}(i,j,k)$$

(3)

$$D_t\mathcal{X}(i,j,k)=\mathcal{X}(i,j,k+1)-\mathcal{X}(i,j,k)$$

(4)

The parameter δ assigns different weights to the time dimension TV and the space dimension TV, allowing for adjustments in the rate of change in the time dimension based on contextual information. We set δ = 0.5 following [73].

3.3. t-SVD and Rank Approximation Based on Laplace Operator

For the traditional tensor robust principal component analysis (TRPCA) model, convex relaxation methods are commonly used to solve traditional tensor robust principal component analysis models. Instead of minimizing the tensor rank, the tensor nuclear norms are minimized. The most commonly used method is t-SVD, which can be described as follows:

$$\mathcal{X}=\mathcal{U}\ast \mathcal{S}\ast {\mathcal{V}}^T$$

(5)

Here, $\mathcal{X}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$ represents the original tensor. $\mathcal{U}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$ and $\mathcal{V}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$ represent the orthogonal tensors after singular value decomposition. $\mathcal{S}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$ is the diagonal tensor. ∗ represents the tensor t-product. The illustration of the Mode-3 t-SVD is shown in Figure 2.

The t-SVD for tensor is presented in Algorithm 1. First, the Fourier transform of the tensor $\mathcal{X}$ is computed along the third dimension to obtain tensor $\bar{\mathcal{X}}$. Singular value decomposition is then performed on each of the n₃ slices ${\bar{\mathcal{X}}}^{(i)}$ of $\bar{\mathcal{X}}$ along the third dimension, resulting in $[\bar{U},\bar{S},\bar{\mathcal{V}}]$. Finally, the tensor result $[\mathcal{U},\mathcal{S},\mathcal{V}]$ is obtained after t-SVD processing by applying an inverse Fourier transform.

Algorithm 1: t-SVD of a 3D Tensor

Input: $\mathcal{X}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$   1. $\bar{\mathcal{X}}=\mathrm{f}\mathrm{f}\mathrm{t}(\mathcal{X},[],3)$   2. for i = 1 to n3 Do   3. $[{\bar{\mathcal{U}}}^{(i)},{\bar{\mathcal{S}}}^{(i)},{\bar{\mathcal{V}}}^{(i)}]=svd({\bar{\mathcal{X}}}^{(i)})$   4. End Do   5. $\mathcal{U}\leftarrow \mathrm{i}\mathrm{f}\mathrm{f}\mathrm{t}(\bar{\mathcal{U}},[],3)$, $\mathcal{S}\leftarrow \mathrm{i}\mathrm{f}\mathrm{f}\mathrm{t}(\bar{\mathcal{S}},[],3)$, $\mathcal{V}\leftarrow \mathrm{i}\mathrm{f}\mathrm{f}\mathrm{t}(\bar{\mathcal{V}},[],3)$   Output: Orthogonal Tensors $\mathcal{U}$ and $\mathcal{V}$, Diagonal Tensor $\mathcal{S}$

Noise and fluctuations in the tensor corresponded to smaller singular values. The t-TNN eliminates these by setting a threshold and then reconstructing a low-rank tensor using the remaining singular values to extract the main information from the tensor. However, T-TNN assigns the same importance to different singular values, which is inconsistent with natural images. Larger singular values convey low-frequency information, whereas smaller singular values convey high-frequency information and should be treated with different weights. Xu et al. [74] introduced a Laplacian operator into a T-TNN, which automatically assigns weights based on the importance of singular values. Its definition is

$$\begin{array}{ll}{\Vert \mathcal{X}\Vert }_{LPTNN}& ={\displaystyle \sum _{k=1}^{n_3}{\displaystyle \sum _i^{\min (n_1,n_2)}\varphi ({\sigma }_i({\bar{\mathcal{X}}}^{(k)}))}}\\ & ={\displaystyle \sum _{k=1}^{n_3}{\displaystyle \sum _i^{\min (n_1,n_2)}\varphi (1-e^{-{\sigma }_i({\bar{\mathcal{X}}}^{(k)})/\epsilon })}}\end{array}$$

(6)

where ε represents a positive number. ${\sigma }_i(•)$ represents the i-th singular value. $\varphi (x)=1-e^{-x/\epsilon }$ represents the Laplace equation. ${\Vert •\Vert }_{LPTNN}$ represents the T-TNN based on the Laplacian operator. Compared to the l₁ norm, the Laplace norm can better approximate the l₀ norm, as shown in Figure 3 [51].

4. Proposed Model

This section provides a detailed description of the proposed TRN denoising bt-LPTVTD model. Figure 4 shows an overview of the model. First, the continuous sequence of images is divided along the temporal domain into tensors of fixed size, Tensor 1 to Tensor n. For each tensor, Tensor k is used as the input to bt-LPTVTD, resulting in the denoised tensor. The bt-LPTVTD method comprises two steps.

First, the input tensor is decomposed into the scene tensor (Scene Tensor 1) and the noise tensor (Noise Tensor 1) using the proposed bidirectional twisted Laplacian truncated nuclear norm (bt-LPTNN) and anisotropic spatiotemporal total variation regularization (ASTTV). Second, the noise tensor is further analyzed using RPCA to extract information. This results in the decomposition of the noise tensor, including Scene Tensor 2 and Noise Tensor 2.

The final output, Denoised Tensor k, is obtained by combining Scene Tensor 1 and Scene Tensor 2. Subsequently, the denoised tensor corresponding to each tensor is obtained individually, thus restoring the image sequence in the time domain. Notably, the processing of each tensor by the denoising bt-LPTVTD model is independent and can be parallelized and synchronized to improve the processing efficiency of the continuous image sequence.

4.1. Bidirectional t-TNN in Spatiotemporal Domain

To facilitate subsequent denoising, the video stream was divided into a series of consecutive fixed-length image sequences. A single image sequence contaminated with noise can be represented using a 3D tensor. The corresponding degradation models are as follows:

$$\mathcal{Y}=\mathcal{X}+\mathcal{N}$$

(7)

where $\mathcal{Y}$, $\mathcal{X}$, and $\mathcal{N}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$ represent the original image sequence tensor, noise-free image sequence tensor, and noisy image sequence tensor, respectively. n₁ and n₂ represent the width and height of a single-frame image, respectively. n₃ represents the number of frames in the image sequence.

Therefore, the denoising task is transformed into separating $\mathcal{X}$ and $\mathcal{N}$ from the original image sequence tensor $\mathcal{Y}$, thereby completing the TRN denoising process. That is,

$$\begin{array}{l}\underset{\mathcal{X},\mathcal{N}}{\arg \min }\mathrm{rank}(\mathcal{X})+\beta {\Vert \mathcal{N}\Vert }_F^2\\ s.t.\mathcal{Y}=\mathcal{X}+\mathcal{N}\end{array}$$

(8)

where $\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{k}(\mathcal{X})$ represents the low-rank approximation of $\mathcal{X}$ and is used to extract the scene information. β represents the regularization coefficient of the noise term, adjusted according to the strength of the noise. ${\Vert \mathcal{N}\Vert }_F^2$ represents the Frobenius norm of the noise tensor $\mathcal{N}$, which is used to suppress TRN in the image sequence tensor.

Compared with single-frame image denoising, the method of constructing tensors using multiple frames has the following advantages:

• Processing image data from a higher dimension allows for better utilization of potential information between image frames.
• By incorporating temporal information, the tensor-based denoising method can effectively suppress noise and improve the denoising results.
• Matrix processing methods such as total variation and low-rank decomposition can be extended to tensors. Many processing methods are available for tensor data.

In general, the image sequence obtained through the image sensor has a strong correlation between frames. Therefore, the constructed tensor exhibits low-rank characteristics, as shown in Figure 5. Singular values of unfolded matrices from an image sequence reflect dimension correlation, where a quicker decline signifies a stronger correlation in that dimension. By utilizing the low-rank properties of spatiotemporal image information, it can be distinguished from TRN and can recover image information from noisy tensors, thus suppressing TRN.

For a tensor $\mathcal{X}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$ composed of a sequence of n₃ frames, the horizontally twisted tensor is defined as $\mathcal{Y}=\overrightarrow{\mathcal{X}},(\mathcal{Y}\in {\mathbb{R}}^{n_1\times n_3\times n_2})$, and the slices of $\mathcal{Y}$ and $\mathcal{X}$ satisfy $\mathcal{Y}(:,:,k)=\mathcal{X}(k,:,:)$. A vertically twisted tensor is defined as $\mathcal{Z}=\overrightarrow{\mathcal{X}},(\mathcal{Z}\in {\mathbb{R}}^{n_2\times n_3\times n_1})$, and the slices of $\mathcal{Z}$ and $\mathcal{X}$ satisfy $\mathcal{Z}(:,:,k)=\mathcal{X}(:,k,:)$. The process of recovering $\mathcal{X}$ from the twisted tensor $\overrightarrow{\mathcal{X}}$ is called squeezing, and the process of tensor twist and squeeze is shown in Figure 6, with the blue arrows indicating the direction of Mode-3.

When a camera records a scene through translation, temporal changes in the image are relatively slow, and the scene information in the image exhibits obvious sparsity. The twisted tensor is more suitable for scenes with global translation and can effectively utilize the inter-frame correlation [62].

Moreover, a bidirectional twisted truncated nuclear norm (bt-TNN) was proposed as an enhancement to the t-TNN to achieve balanced noise reduction across various spatial directions. The difference between bt-TNN and t-TNN lies in the alternating use of horizontally twisted and vertically twisted tensors during the iterative denoising process, which enhances the ability of the algorithm to suppress noise in both the horizontal and vertical directions.

Based on the above analysis, the temporal noise suppression process of the image sequence can be represented as

$$\begin{array}{l}\underset{\mathcal{X}.\mathcal{N}}{\arg \min }{\Vert \mathcal{X}\Vert }_{bt-TNN}+\beta {\Vert \mathcal{N}\Vert }_F^2\\ s.t.\mathcal{Y}=\mathcal{X}+\mathcal{N}\end{array}$$

(9)

where ${\Vert •\Vert }_{bt-TNN}$ represents the bidirectional twisted truncated nuclear norm.

4.2. Tensor Decomposition Based on Bidirectional Twisted Laplacian Nuclear Norm and Spatiotemporal Total Variation

By combining the regularization of the ASTTV with the low-rank approximation based on the bidirectional twisted tensor Laplacian nuclear norm, the model can be updated as

$$\begin{array}{ll}\mathcal{X},\mathcal{N}=& \underset{\mathcal{X},\mathcal{N}}{\arg \min } {\Vert \mathcal{X}\Vert }_{\mathrm{bt}-LPTNN}+\lambda {\Vert \mathcal{X}\Vert }_{ASTTV}+\beta {\Vert \mathcal{N}\Vert }_F^2\\ & s.t.\mathcal{Y}=\mathcal{X}+\mathcal{N}\end{array}$$

(10)

where ${\Vert •\Vert }_{bt-LPTNN}$ represents bt-TNN based on the Laplacian operator. λ represents the regularization parameter of the ASTTV regularization term. ${\Vert •\Vert }_{bt-LPTNN}$ can effectively extract the background information from an image sequence. ${\Vert •\Vert }_{ASTTV}$ is used to extract spatiotemporal information in the image sequence and can effectively preserve local detailed information, while eliminating noise.

By substituting (1), (10) can be further expressed as

$$\begin{array}{ll}\mathcal{X},\mathcal{N}=& \underset{\mathcal{X},\mathcal{N}}{\arg \min } {\Vert \mathcal{X}\Vert }_{bt-LPTNN}+\beta {\Vert \mathcal{N}\Vert }_F^2\\ & +\lambda ({‖D_h(\mathcal{X})‖}_1+{‖D_v(\mathcal{X})‖}_1+\delta {‖D_t(\mathcal{X})‖}_1)\\ & s.t.\mathcal{Y} =\mathcal{X}+\mathcal{N}\end{array}$$

(11)

4.3. Spatial Detail Recovery from Noise via RPCA

Combining the 3D tensor low-rank decomposition method with total variation yields better results for recovering the motion component. However, the detailed information of the image is still lost, manifested as prominent edges and textures in the noise tensor. Using (11) as the denoising model, the CDNET dataset [75] with Gaussian noise added to the pedestrian scene is used as the input tensor. The resulting denoised and noisy images are presented in Figure 7.

From the perspective of the time domain, the edges and textures in the noise tensor $\mathcal{N}$ remain relatively unchanged. Therefore, principal component analysis (PCA), a statistical method that identifies the most significant patterns in data by reducing their dimensionality, can be used to further extract detailed information. For the purpose of extracting detailed information while maintaining computational efficiency, the dimensionality of the noise tensor $\mathcal{N}$ is initially reduced, followed by the construction of a 2D matrix $N_0$. The first dimension of $N_0$ represents the spatial domain, and the second dimension represents the time domain. Then, we use RPCA [76] to decompose $N_0$ into a low-rank matrix $T$ and a noise matrix $N_1$. Finally, $T$ and $N_1$ are restored to 3D tensors $\mathcal{T}$ and $\mathcal{M}$, respectively. $\mathcal{M}$ is the final noise tensor, and $\mathcal{X}$ + $\mathcal{T}$ is the final denoised image sequence tensor.

By further extracting detailed information from the noisy tensor $\mathcal{N}$ through RPCA, the TRN extraction model was improved to preserve detailed information in the image, as illustrated in Figure 8. The RPCA process is described as follows.

$$\begin{array}{c}T,N_1=\underset{\mathbf{T},{\mathbf{N}}_{\mathbf{1}}}{\mathrm{a}\mathrm{r}\mathrm{g} \mathrm{m}\mathrm{i}\mathrm{n}}{\Vert T\Vert }_{*}+{\lambda }_N{\Vert N_1\Vert }_F^2\\ s.t.N_0=T+N_1\end{array}$$

(12)

Here, $N_0$ is constructed by reducing the dimensions of $\mathcal{N}$. $T$ and $N_1$ are low-rank and noise matrices, respectively, obtained through RPCA decomposition. λ_N is a regularization parameter of the noise term.

The improved denoising model is composed of Equations (11) and (12), which correspond to the first and second steps of bt-LPTVTD, respectively.

In summary, the proposed model aims to suppress the TRN. First, bt-LPTNN was used to extract the background information of the image sequence by combining the Laplace operator and tensor truncation nuclear norm. The Laplace operator can automatically allocate weight coefficients to singular values based on image information and effectively preserve the structural information. Simultaneously, ASTTV is used to further extract the structure and detailed information of the image, and it has a good denoising capability. In addition, after the dimensionality reduction of the noise tensor, RPCA is used to extract the low-rank part, which can better preserve detailed information.

4.4. Optimization Procedure

For the proposed bt-LPTVTD denoising model, (12) is solved using [76], whereas (11) utilizes the Alternating Direction Method of Multipliers (ADMM) framework [77]—an iterative algorithm designed for solving convex optimization problems, particularly those involving separable objective functions—to solve the bt-LPTVTD model as a robust tensor recovery problem. By introducing the auxiliary variables $\mathcal{Z}$, ${\mathcal{V}}_1=D_h(\mathcal{X})$, ${\mathcal{V}}_2=D_v(\mathcal{X})$, and ${\mathcal{V}}_3=D_t(\mathcal{X})$, Equation (11) can be written as

$$\begin{gathered} \mathcal{X},\mathcal{N}=\underset{\mathcal{X},\mathcal{N}}{\arg \min }{\Vert \mathcal{Z}\Vert }_{bt-LPTNN}+\beta {\Vert \mathcal{N}\Vert }_F^2+\lambda ({\Vert {\mathcal{V}}_1\Vert }_1+{\Vert {\mathcal{V}}_2\Vert }_1+\delta {\Vert {\mathcal{V}}_3\Vert }_1) \\ s.t.\mathcal{Y}=\mathcal{X}+\mathcal{N},\mathcal{Z}=\mathcal{X},{\mathcal{V}}_1=D_h(\mathcal{X}),{\mathcal{V}}_2=D_v(\mathcal{X}),{\mathcal{V}}_3=D_v(\mathcal{X}) \end{gathered}$$

(13)

Equation (13) can be transformed into an augmented Lagrange function,

$$\begin{array}{ll}{L}_A& (\mathcal{X},\mathcal{N},\mathcal{Z},\mathcal{F},{\mathcal{V}}_1,{\mathcal{V}}_2,{\mathcal{V}}_3)\\ & ={\Vert \mathcal{Z}\Vert }_{bt-LPTNN}+\lambda ({‖{\mathcal{V}}_1‖}_1+{‖{\mathcal{V}}_2‖}_1+\delta {‖{\mathcal{V}}_3‖}_1)+\beta {\Vert \mathcal{N}\Vert }_F^2\\ & +〈M_1,\mathcal{X}-\mathcal{Y}-\mathcal{N}〉+〈M_2,\mathcal{Z}-\mathcal{X}〉+〈M_3,{\mathcal{V}}_1-D_h(\mathcal{X})〉\\ & +〈M_4,{\mathcal{V}}_2-D_v(\mathcal{X})〉+〈M_5,{\mathcal{V}}_3-D_t(\mathcal{X})〉\\ & +{\displaystyle \frac{\mu }{2}}({‖\mathcal{Y}-\mathcal{X}-\mathcal{N}‖}_F^2+{‖\mathcal{Z}-\mathcal{X}‖}_F^2+{‖{\mathcal{V}}_1-D_h(\mathcal{X})‖}_F^2\\ & +{‖{\mathcal{V}}_2-D_v(\mathcal{X})‖}_F^2+{‖{\mathcal{V}}_3-D_t(\mathcal{X})‖}_F^2)\end{array}$$

(14)

where μ represents the positive penalty constant. M₁, M₂, M₃, M₄, and M₅ represent Lagrange multipliers. The ADMM algorithm is used to divide Equation (14) into six subproblems and solve them through iterative updates.

(1) Updating $\mathcal{Z}$ by fixing the other variables,

$${\mathcal{Z}}^{k+1}=\underset{\mathcal{Z}}{\arg \min }{\Vert \mathcal{Z}\Vert }_{bt-LPTN}+{\displaystyle \frac{{\mu }^k}{2}}{\Vert \mathcal{Z}-{\mathcal{X}}^k+{\displaystyle \frac{M_2^k}{{\mu }^k}}\Vert }_F^2$$

(15)

For optimization problems of the following form,

$$\underset{\mathcal{Z}}{\arg \min }{\Vert \mathcal{Z}\Vert }_{bt-LPTN}+{\displaystyle \frac{\eta }{2}}{\Vert \mathcal{Z}-\mathcal{Q}\Vert }_F^2,$$

(16)

Algorithm 2 provides the solution for each iteration [51].

Algorithm 2: ADMM of (16)

Input: $\mathcal{Q}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$, η, ε   Output: ${\bar{\mathcal{Z}}}^{k+1}$, ${\mathcal{Z}}^{k+1}$   Step 1: $\bar{\mathcal{Q}}=\mathrm{f}\mathrm{f}\mathrm{t}(\mathcal{Q},[],3)$   Step 2: Calculate the ${\bar{\mathcal{Z}}}^{k+1}$ for each temporal slice through the following process.   for $l=1,\cdots ,[n_3+1/2]$ do    1. $[{\bar{\mathcal{U}}}^{(l)},{\bar{\mathcal{S}}}^{(l)},{\bar{\mathcal{V}}}^{(l)}]=svd({\bar{\mathcal{Q}}}^{(l)})$    2. ${\bar{D}}_{\nabla \varphi /\eta }^l={({\bar{S}}^l(i,j,k)-\nabla \varphi ({\sigma }_i^{k,l})/\eta )}_{+}$    3. $({\bar{\mathcal{Z}}}^{k+1}{)}^l={\bar{U}}^l\ast {\bar{D}}_{\nabla \varphi /\eta }^l\ast ({\bar{V}}^l{)}^H$   end for   for $l=[n_3+1/2],\cdots ,1$ do    $({\bar{\mathcal{Z}}}^{k+1}{)}^l=\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{j}(({\bar{\mathcal{Z}}}^{k+1}{)}^{(n_3-l+2)})$   end for   Step 3: Calculate $({\mathcal{Z}}^{k+1})=\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{t}({\bar{\mathcal{Z}}}^{k+1},[],3)$

Let ${\overrightarrow{\mathcal{X}}}^k-{\displaystyle \frac{M_2^k}{{\mu }^k}}=\mathcal{U}\ast \mathcal{S}\ast {\mathcal{V}}^H$, and the optimal result of (14) can be obtained using Algorithm 2,

$${\mathcal{Z}}^{k+1}=\mathcal{U}\ast {\mathcal{D}}_{\nabla \varphi /\alpha }\ast {\mathcal{V}}^H,$$

(17)

where ${\bar{D}}_{\nabla \varphi /\alpha }={\left(\bar{S}(i,j,k)-{\displaystyle \frac{\nabla \varphi ({\sigma }_i^{k,l})}{\alpha }}\right)}_{+}$.

(2) Updating $\mathcal{X}$ by fixing the other variables,

$$\begin{array}{ll}{\mathcal{X}}^{k+1}=& \underset{\mathcal{X}}{\arg \min }{\displaystyle \frac{{\mu }^k}{2}}({\Vert \mathcal{Y}-\mathcal{X}-{\mathcal{Y}}^k+{\displaystyle \frac{M_1^k}{{\mu }^k}}\Vert }_F^2+{\Vert Z^{k+1}-\mathcal{X}+{\displaystyle \frac{M_2^k}{{\mu }^k}}\Vert }_F^2\\ & +{\Vert {\dot{\mathcal{V}}}_1^k-D_h(\mathcal{X})+{\displaystyle \frac{M_3^k}{{\mu }^k}}\Vert }_F^2+{\Vert {\dot{\mathcal{V}}}_2^k-D_v(\mathcal{X})+{\displaystyle \frac{M_4^k}{{\mu }^k}}\Vert }_F^2\\ & +{\Vert {\dot{\mathcal{V}}}_3^k-D_t(\mathcal{X})+{\displaystyle \frac{M_5^k}{{\mu }^k}}\Vert }_F^2)\end{array}$$

(18)

The optimization problem in (18) can be viewed as the following linear system:

$$(2\mathrm{I}+\Delta ){\mathcal{X}}^{k+1}=L_1^k+{\theta }_1+{\theta }_2+{\theta }_3$$

(19)

where $\Delta =D_h^T{D}_h+D_v^T{D}_v+D_t^T{D}_t$, $L_1^k=\mathcal{Y}-{\mathcal{N}}^k+{\displaystyle \frac{M_1^k}{{\mu }^k}}+{\mathcal{Z}}^{k+1}+{\displaystyle \frac{M_2^k}{{\mu }^k}}$, ${\theta }_1=D_h^T(V_1^k+{\displaystyle \frac{M_3^k}{{\mu }^k}})$, ${\theta }_2=D_v^T(V_2^k+{\displaystyle \frac{M_4^k}{{\mu }^k}})$, ${\theta }_3=D_t^T(V_3^k+{\displaystyle \frac{M_5^k}{{\mu }^k}})$. ${(•)}^T$ represents the transposition of the matrix. By considering $D_h(\mathcal{X})$, $D_v(\mathcal{X})$, and $D_t(\mathcal{X})$ as convolutions in three directions, the closed-form solution to (18) can be derived.

$${\mathcal{X}}^{k+1}={\mathrm{n}\mathrm{F}\mathrm{F}\mathrm{T}}^{-1}\left({\displaystyle \frac{\mathrm{n}\mathrm{F}\mathrm{F}\mathrm{T}(L_1^k+{\theta }_1+{\theta }_2+{\theta }_3)}{2+{\sum }_{i\in \{h,v,t\}}\mathrm{n}\mathrm{F}\mathrm{F}\mathrm{T}(D_i{)}^H\mathrm{n}\mathrm{F}\mathrm{F}\mathrm{T}(D_i)}}\right)$$

(20)

where H, nFFT, and nFFT-1 denote the complex conjugate, fast nFFT operator, and inverse nFFT operator, respectively.

(3) Updating ${\mathcal{V}}_1$, ${\mathcal{V}}_2$, and ${\mathcal{V}}_3$ by fixing the other variables,

$$\left\{\begin{array}{c}{\mathcal{V}}_1^{k+1}=\underset{{\mathcal{V}}_1}{\arg \min } \lambda {‖{\mathcal{V}}_1‖}_1+{\displaystyle \frac{{\mu }^k}{2}}{‖{\mathcal{V}}_1-D_h({\mathcal{X}}^{k+1})+{\displaystyle \frac{M_3^k}{{\mu }^k}}‖}_F^2\\ {\mathcal{V}}_2^{k+1}=\underset{{\mathcal{V}}_2}{\arg \min } \lambda {‖{\mathcal{V}}_2‖}_1+{\displaystyle \frac{{\mu }^k}{2}}{‖{\mathcal{V}}_2-D_v({\mathcal{X}}^{k+1})+{\displaystyle \frac{M_4^k}{{\mu }^k}}‖}_F^2\\ {\mathcal{V}}_3^{k+1}=\underset{{\mathcal{V}}_3}{\arg \min } \delta \lambda {‖{\mathcal{V}}_3‖}_1+{\displaystyle \frac{{\mu }^k}{2}}{‖{\mathcal{V}}_3-D_t({\mathcal{X}}^{k+1})+{\displaystyle \frac{M_5^k}{{\mu }^k}}‖}_F^2\end{array}\right.$$

(21)

The above problem can be solved using the element-wise shrinkage operator as

$$\{\begin{array}{c}{\mathcal{V}}_1^{k+1}=T{h}_{\lambda /{\mu }^k}(D_{\mu }{\mathcal{X}}^{k+1}-{\displaystyle \frac{M_3^k}{{\mu }^k}})\\ {\mathcal{V}}_2^{k+1}=T{h}_{\lambda /{\mu }^k}(D_v{\mathcal{X}}^{k+1}-{\displaystyle \frac{M_4^k}{{\mu }^k}})\\ {\mathcal{V}}_3^{k+1}=T{h}_{\delta \mu /{\mu }^k}(D_t{\mathcal{X}}^{k+1}-{\displaystyle \frac{M_5^k}{{\mu }^k}})\end{array}$$

(22)

(4) Updating $\mathcal{N}$ by fixing the other variables,

$${\mathcal{N}}^{k+1}=\underset{\mathcal{N}}{\arg \min } \beta \Vert \mathcal{N}{\Vert }_F^2+{\displaystyle \frac{{\mu }^k}{2}}\Vert \mathcal{Y}-{\mathcal{X}}^{k+1}-\mathcal{N}+{\displaystyle \frac{M_1^k}{{\mu }^k}}{\Vert }_F^2$$

(23)

$${\mathcal{N}}^{k+1}={\displaystyle \frac{{\mu }^k(\mathcal{Y}-{\mathcal{X}}^{k+1})+M_1^k}{2\beta +{\mu }^k}}$$

(24)

(5) Updating M₁, M₂, M₃, M₄, and M₅ by fixing the other variables,

$$\left\{\begin{array}{l}{M}_1^{k+1}=M_1^k+{\mu }^k(\mathcal{Y}-{\mathcal{X}}^{k+1}-{\mathcal{N}}^{k+1})\\ M_2^{k+1}=M_2^k+{\mu }^k({\mathcal{X}}^k-{\mathcal{Z}}^k)\\ M_3^{k+1}=M_3^k+{\mu }^k({\mathcal{V}}_1^{k+1}-D_h({\mathcal{X}}^{k+1}))\\ M_4^{k+1}=M_4^k+{\mu }^k({\mathcal{V}}_2^{k+1}-D_v({\mathcal{X}}^{k+1}))\\ M_5^{k+1}=M_5^k+{\mu }^k({\mathcal{V}}_3^{k+1}-D_t({\mathcal{X}}^{k+1}))\end{array}\right.$$

(25)

(6) Updating μ by fixing the other variables,

$${\mu }^{k+1}=\min (\rho {\mu }^k,{\mu }_{\max })$$

(26)

The proposed bt-LPTVTD denoising algorithm is summarized in Algorithm 3.

Algorithm 3: bt-LPTVTD TRN denoise algorithm

Input: Image sequence $d_1,d_2,\cdots ,d_{n_3}\in {\mathbb{R}}^{n_1\times n_2}$, The number of images, n3, for building tensors, parameters λ, β, and μ greater than 0.   Output: Denoised Tensor $\mathcal{X}$ + $\mathcal{T}$, Noise Tensor $\mathcal{M}$.   1: Build tensor $\mathcal{Y}$ from image sequence.   2: Initialize: ${\mathcal{X}}^0={\mathcal{N}}^0={\mathcal{V}}_1^0={\mathcal{V}}_2^0={\mathcal{V}}_3^0=0$, $M_i^0=0$, i = 1, 2, …, 5, ${\mu }^0=0.01$, ${\mu }^{\mathrm{m}\mathrm{a}\mathrm{x}}=1\times 10^6$, k = 0, $\rho =1.5$, $\epsilon =1\times 10^{-6}$.   3: While: not convergence do   4: Calculate the twisted tensor $\overrightarrow{\mathcal{Y}}=twist(\mathcal{Y})$. Use the horizontally twisted tensor when k is odd, and use the vertically twisted tensor when k is even.   5: Using Algorithm 2 to update ${\overrightarrow{\mathcal{Z}}}^{k+1}$ and obtain squeezing tensor ${\mathcal{Z}}^{k+1}=squeeze({\overrightarrow{\mathcal{Z}}}^{k+1})$.   6: Update ${\mathcal{X}}^{k+1}$ using Formula (20).   7: Update ${\mathcal{V}}_1$, ${\mathcal{V}}_2$ and ${\mathcal{V}}_3$ using Formula (22).   8: Update $\mathcal{N}$ using Formula (24).   9: Update M1, M2, M3, M4, and M5 using Formula (25).   10: Update μ using Formula (26).   11: Check the convergence conditions $\Vert \mathcal{Y}-{\mathcal{X}}^{k+1}-{\mathcal{N}}^{k+1}{\Vert }_F^2/\Vert \mathcal{Y}{\Vert }_F^2\le \epsilon$   12: k = k + 1.   13: end while   14: Construct $N_0$ by reducing the dimensionality of $\mathcal{N}$, separating the spatial and temporal domains into different dimensions.   15: Use RPCA to decompose $N_0$ into $T$ and $N_1$.   16: Restore $T$ and $N_1$ to 3D tensors $\mathcal{T}$ and $\mathcal{M}$, respectively.

4.5. Image Sequence Denoising Procedure

The core steps of the bt-LPTVTD model-based TRN denoising method are as follows:

(1)

Tensor Construction: For an image sequence $d_1,d_2,\cdots ,d_{n_3},\cdots \in {\mathbb{R}}^{n_1\times n_2}(n_3\ge 1)$, $n_3$ consecutive images are combined to form the original image sequence tensor, $\mathcal{Y}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$. For best results, $n_3$ is recommended to be between 30 and 50, balancing denoising and computation.

(2)

Algorithm 3 is used to decompose the original image sequence tensor $\mathcal{Y}$ into denoised image sequence tensor $\mathcal{X}$ and noise tensor $\mathcal{N}$.

(3)

Detail Extraction: RPCA is used to decompose $\mathcal{N}$ into a low-rank tensor $\mathcal{T}$ and a noise tensor $\mathcal{M}$, thereby obtaining the denoised image sequence tensor $\mathcal{N}$ + $\mathcal{T}$ and completing the temporal denoising of n₃ consecutive images.

(4)

Iterative Processing: The next $n_3$ consecutive images are combined into a new tensor. The above steps are then repeated to denoise the continuous image data flow block-wise. It is important to note that adjacent tensors are processed using the same set of parameters, ensuring consistency across the entire image sequence (this is valid when the noise intensity and scene information approximately constant).

4.6. Complexity Analysis

For the input image sequence $d_1,d_2,···,d_L\in {\mathbb{R}}^{n_1\times n_2}(L\ge 1)$, a maximum of $L/n_3$ tensors $\mathcal{X}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$ can be obtained. The computation cost of bt-LPTVTD is mainly derived from the matrix SVD and the fast Fourier transform (FFT).

Updating $\mathcal{Z}$ requires performing FFT and $⌈{\displaystyle \frac{(n_2+1)}{4}}⌉$ + $⌈{\displaystyle \frac{(n_1+1)}{4}}⌉$ SVDs of $n_1\times n_3$ and $n_2\times n_3$ matrices in each iteration, which cost

$$\mathcal{O}({\displaystyle \frac{L{n}_3}{2}}A+({\displaystyle \frac{L{n}_3^2}{n_1{n}_2}}\left)B\right)$$

(27)

where $A=n_1\log n_2+n_2\log n_1$ and $B={n_1}^2⌈{\displaystyle \frac{(n_2+1)}{4}}⌉+{n_2}^2⌈{\displaystyle \frac{(n_1+1)}{4}}⌉$.

Updating $\mathcal{X}$ requires performing an FFT operation, which costs $\mathcal{O}(L{n}_1{n}_2(\log n_3))$. RPCA costs $\mathcal{O}(L{n}_1{n}_2)$.

In summary, the computational complexity is

$$\mathcal{O}\left({\displaystyle \frac{kL{n}_3}{2}}A+kL{n}_1{n}_2\left(1+\log n_3\right)+\left({\displaystyle \frac{L{n_3}^2}{n_1{n}_2}}\right)B\right)$$

(28)

where k denotes the iteration times.

4.7. Convergence Analysis

Xu et al. [74] proved that the LPTNN under the t-SVD algorithm is a nonconvex optimization problem. The proposed algorithm converges, and the optimal solution can be computed using the ADMM framework and the convergent objective function. The convergence of the bt-LPTVTD model can be observed by examining multiple image sequence examples. The iteration curve (as shown in Figure 9) converges, and the objective function changes slowly after approximately 20 iterations.

5. Experimental Results and Analyses

In this section, the performance of the bt-LPTVTD algorithm is tested using both simulated and real data. The proposed bt-LPTVTD algorithm was compared to six commonly used algorithms: VBM4D [40], RPCA [76], LRTDTV [49], LRTDGS [64], ASTTV-NTLA [51], and SRSTT [50]. Among these, VBM4D and RPCA are extensions from 2D to 3D, LRTDTV and LRTDGS are algorithms for HSI denoising, and ASTTV-NTLA and SRSTT are algorithms for small-target recognition. The code for these six algorithms was obtained from the authors’ open-source repositories. The initial parameter settings were based on the authors’ recommendations and then fine-tuned based on the denoising performance to obtain the best denoising effect for each algorithm under various noise conditions. The initial parameter settings and the denoised output data of the seven algorithms are listed in

Table 2. Parameter settings for comparative algorithms.

Comparative Algorithms		Parameter Settings	Output Data
2D algorithm expansion	VBM4D	profile = ‘np’, do_wiener = 0, sharpen = 1, deflicker = 0.5, verbose = 1, est_noise = 1	Denoised Tensor $\mathcal{Y}$
	RPCA	λ = 0.003	Denoised Tensor $\mathcal{L}$, Noise Tensor $\mathcal{S}$
Hyperspectral image denoising algorithms	LRTDTV	τ = 1, λ = 100,000, β = 100	Denoised Tensor $\mathcal{X}$, Noise Tensor $\mathcal{S}$ + $\mathcal{N}$
	LRTDGS	λ1 = 0.8, λ2 = 10	Denoised Tensor $\mathcal{X}$, Noise Tensor $\mathcal{S}$ + $\mathcal{N}$
Small object detection algorithms	ASTTV-NTLA	λtv = 0.01, λs = 0.1, λ3 = 0.5	Denoised Tensor $\mathcal{B}$ + $\mathcal{T}$, Noise Tensor $\mathcal{N}$
	SRSTT	λ1 = 10,000, λ2 = 0.1, λ3 = 0.5	Denoised Tensor $\mathcal{B}$ + $\mathcal{T}$, Noise Tensor $\mathcal{N}$
The proposed algorithm	bt-LPTVTD	λ = 0.0005, β = 0.01, λN = 0.003	Denoised Tensor $\mathcal{X}$ + $\mathcal{T}$, Noise Tensor $\mathcal{M}$

. Before processing, all images were preprocessed to normalize the grayscale values to the range of [0,1] and then restored to the original grayscale levels after denoising. All the algorithms were run on the same hardware and software platform to compare their efficiencies.

5.1. Simulation Experiments

5.1.1. Experimental Settings

In accordance with the scene requirements for image target detection, three datasets were identified and subjected to separate tests on the proposed bt-LPTVTD method for denoising the TRN in the infrared band under three scenarios: static, moving-target, and moving-camera scenes. The datasets were captured using the Xcore LA6110 model vanadium-oxide uncooled long-wave infrared focal plane detector (IRay Technology Co., Ltd., Yantai, China). The detector exhibited a frame rate of 50 Hz, a resolution of 512 × 640, and a spectral response range of 8–14 μm. The acquired image sequence was mapped to grayscale values in the range [0, 1] and downsampled to a size of 256 × 320, forming infrared datasets. Downsampling was performed by extracting pixels in odd-numbered rows and columns to improve subsequent processing efficiency.

The infrared datasets consisted of the following three parts: (1) Infrared Dataset 1 was a static scene in which the entire scene remained within the temporal domain. (2) Infrared Dataset 2 consisted of scenes with moving targets, with only the pedestrian portion exhibiting motion. (3) Infrared Dataset 3 was captured while the camera was moving horizontally at a constant speed to capture the uniform motion of the entire scene.

To simulate various noise conditions in infrared images, a detailed subjective and objective evaluation of the algorithm’s denoising performance was performed by introducing pointwise and stripewise Gaussian noise. The specifics are as follows.

Case 1: For different scenes in the infrared datasets, stripewise Gaussian noise was added to each frame of the image sequence with a mean of zero and variance of 0.01. The number of stripes for row and column noise was set to 0.3 times the resolution of the image row and column, respectively.

Case 2: For different scenes in the infrared datasets, we added pointwise and stripewise Gaussian noise to each frame of the image sequence. The mean of the pointwise Gaussian noise was 0 and the variance was 0.005. The addition of stripewise Gaussian noise was the same as that in Case 1.

To evaluate the denoising performance of all the methods, two metrics, namely the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), were used in the simulation experiment. PSNR and SSIM are widely used metrics for evaluating image quality after denoising, as they effectively measure the reduction of noise and the preservation of image structure. PSNR quantifies the difference between the original and denoised images based on the mean squared error (MSE), while SSIM assesses the perceived structural similarity. Higher values of both metrics generally indicate better denoising performance. The parameters of all the comparative algorithms were adjusted to maximize the PSNR and SSIM.

The PSNR represents the ratio of the maximum possible power of a signal to the power of the noise and is given by

$$\mathrm{PSNR}=10{\log }_{10}\left({\displaystyle \frac{L^2}{\mathrm{MSE}}}\right)$$

(29)

where L represents the maximum possible value of the grayscale. Because we normalized the grayscale values to [0, 1], L was set to one. MSE is the mean squared error, which is given by

$$\mathrm{MSE}={\displaystyle \frac{1}{mn}}{\displaystyle \sum }_{i=0}^{m-1}{\displaystyle \sum }_{j=0}^{n-1}{[X(i,j)-Y(i,j)]}^2$$

(30)

where m and n are the image dimensions. $X(i,j)$ and $Y(i,j)$ represent the grayscale values of the i-th row and j-th column of the two images, respectively.

SSIM measures the structural similarity between two images, which is given by

$$\mathrm{SSIM}(X,Y)={\displaystyle \frac{(2{\mu }_X{\mu }_Y+c_1)(2{\sigma }_{XY}+c_2)}{({\mu }_X^2+{\mu }_Y^2+c_1)({\sigma }_X^2+{\sigma }_Y^2+c_2)}}$$

(31)

where X and Y denote the two images to be compared. ${\mu }_X$ and ${\mu }_Y$ represent the mean values of the images X and Y, respectively. ${\sigma }_X^2$ and ${\sigma }_Y^2$ represent the variances in images X and Y, respectively. ${\sigma }_{XY}$ represents the covariance of images X and Y. c₁ and c₂ are constants used to avoid division by zero.

5.1.2. Subjective Evaluation

To subjectively compare the effectiveness of the algorithms, several scenarios from Cases 1 and 2 were selected to compare the denoising results of different algorithms on the same frames of Infrared Dataset 2, as shown in Figure 10. Critical regions were magnified using bilinear interpolation to facilitate visual comparison of denoising effects across various algorithms. For the pointwise and stripewise Gaussian noise in the infrared datasets in Cases 1 and 2, VBM4D, LRTDTV, LRTDGS, ASTTV-NTLA, and SRSTT were unable to effectively suppress stripewise Gaussian noise, and only VBM4D did not blur the moving person. RPCA and bt-LPTVTD were able to effectively suppress both pointwise and stripewise Gaussian noise. However, RPCA completely lost motion information. This is because it flattens the 3D video data into a 2D matrix, destroying the spatial relationships within frames and making motion indistinguishable from noise [76]. In contrast, bt-LPTVTD was able to preserve scene information of both static and moving parts after denoising.

5.1.3. Quantitative Evaluation

The subjective evaluation demonstrates the effectiveness of the bt-LPTVTD method. Subsequently, the objective evaluation metrics of PSNR and SSIM were used to quantitatively analyze the performance of the bt-LPTVTD method in all the simulated experiments, representing the average values of the computed results for all frames in the selected image sequences. Generally, higher PSNR and SSIM values indicate better denoising.

Table 3. Quantitative comparisons of all the comparative methods for infrared datasets under different noise conditions (Bold = Best PSNR/SSIM; Underline = Second-best PSNR/SSIM).

Datasets	Cases	Index	Noisy	VBM4D	RPCA	LRTDTV	LRTDGS	ASTTV-NTLA	SRSTT	bt-LPTVTD
Infrared Dataset 1 (Static)	Case 1	PSNR	21.2587	21.9832	28.1573	24.6760	26.8218	23.8302	22.6063	26.7460
		SSIM	0.6020	0.5907	0.9083	0.8306	0.8169	0.6654	0.6123	0.8754
	Case 2	PSNR	19.9059	22.5059	23.3729	24.6535	23.7046	22.4778	22.6332	25.1003
		SSIM	0.4713	0.6059	0.8961	0.8226	0.7815	0.6049	0.6115	0.8707
Infrared Dataset 2 (Moving Target)	Case 1	PSNR	21.1069	21.9183	25.7344	24.6713	26.7003	24.5505	22.6308	27.0484
		SSIM	0.4859	0.4851	0.8728	0.7748	0.7640	0.6876	0.5340	0.8358
	Case 2	PSNR	19.9271	22.6795	22.2955	24.6825	23.7482	22.5716	22.6049	25.3411
		SSIM	0.3607	0.5257	0.8602	0.7723	0.7087	0.5393	0.5373	0.8858
Infrared Dataset 3 (Moving Camera)	Case 1	PSNR	21.2205	21.8206	20.8103	23.0913	24.1542	23.7634	21.5072	24.8054
		SSIM	0.5051	0.5157	0.5314	0.6505	0.6100	0.7084	0.4923	0.7374
	Case 2	PSNR	19.8845	22.6072	19.7699	23.0571	22.6151	21.5046	21.4709	23.0607
		SSIM	0.3751	0.5646	0.5095	0.6388	0.5797	0.4912	0.4885	0.6556

shows the evaluation results of the PSNR and SSIM metrics for all the comparison methods under the Case 1 and Case 2 conditions. The best and second-best results among all the comparison methods are highlighted in bold and underlined, respectively, for each metric.

In Cases 1 and 2 of the infrared datasets, the denoising effects of stripewise Gaussian TRN and pointwise TRN were tested under static, moving-target, and moving-camera conditions. While RPCA exhibits superior performance in both static scenes and scenarios with strong mixed noise, primarily attributed to its precise SVD decomposition and robust constraints on sparsity, it sacrifices motion information, which has a minimal impact in these specific contexts. bt-LPTVTD, on the other hand, employs an approximate solution and compromises noise reduction capabilities to retain motion information. This results in a performance inferior to that of RPCA in those two scenarios. However, considering a broader range of scenarios, including those with target motion as observed during surveillance, bt-LPTVTD demonstrates its overall superiority. It effectively completes the denoising task while preserving crucial details in both the static and moving parts of the scene across these cases.

VBM4D exhibits robust denoising stability, maintaining consistent performance in the presence of both stripe and mixed Gaussian noise, even with increased motion areas. However, its PSNR and SSIM metrics are not outstanding. Combined with the results of subjective evaluation, this is likely due to its insufficient suppression of stripe noise. The hyperspectral denoising methods LRTDTV and LRTDGS achieve good performance in background regions. However, their capability drops sharply as the motion area increases. This is consistent with the subjective evaluation, which indicates that the background is well recovered while the human target is blurred. In contrast, the small-target detection methods ASTTV-NTLA and SRSTT are insensitive to the variation in the motion area, but their overall noise suppression ability is limited. Subjective evaluation reveals an overall blurriness in the image, confirming their poor denoising performance.

Figure 11 shows the PSNR and SSIM performance of each method on the infrared datasets. bt-LPTVTD achieves the highest median values in both metrics and has the most compact boxes, indicating its superior and stable performance. In general, using a combination of a twisted tensor model, ASTTV regularization, and a low-rank approximation based on the Laplacian operator, bt-LPTVTD can effectively suppress the TRN while preserving the detailed information of the moving parts. This was validated through simulated experiments using infrared datasets. Moreover, using RPCA on the noise tensor, in contrast to using it directly on the original image tensor, helps to extract finer details within the temporally stationary parts. This significantly improves bt-LPTVTD’s ability to restore static image regions. Benefiting from the aforementioned approach, bt-LPTVTD demonstrates excellent performance in scenarios with partial motion, such as those containing moving targets like pedestrians, and is expected to have a significant beneficial impact in target detection applications for scene monitoring.

5.2. Real Experiments

The equipment used to collect the infrared datasets for real experiments is identical to that used in the simulation experiments. In contrast to the simulation experiment, an overcast and foggy monitoring scenario was deliberately chosen to introduce more pronounced TRN into the collected image sequence, which includes both temporally static and moving components. This allows us to assess the ability of various methods to recover detailed textures and moving targets while suppressing temporal noise.

The denoising results for the infrared datasets are shown in Figure 12, where the yellow vertical lines indicate the vertical slices displayed on the right side of the yellow box. Figure 12a shows that the infrared image exhibits weak pointwise temporal noise. This noise is primarily composed of thermal noise (Gaussian distributed), 1/f noise, and generation-recombination noise, as described in [10] for uncooled IRFPAs. All denoising algorithms were able to improve the noise. However, bt-LPTVTD and VBM4D showed the best visual effects, preserving the clarity of the moving targets while reducing noise. As shown in Figure 12b, the infrared image was contaminated by strong pointwise temporal noise. bt-LPTVTD not only effectively removed noise but also preserved the point target within the red box.

5.3. bt-LPTVTD for Visible Image Denoising

To further demonstrate bt-LPTVTD’s ability to preserve details in static regions and avoid artifacts in dynamic regions, we conducted experiments on visible images. These images, with rich texture and controllable motion (e.g., Figure 13 and Figure 14), allow for clear visual assessment of these denoising aspects, complementing the infrared results. The visible datasets were captured using a Hikvision (Hangzhou, China) industrial camera, MV-CS060-10UM, with a frame rate of 50 Hz, producing grayscale videos. To enhance experimental efficiency, the videos were downsampled to 256 × 320 pixels by extracting odd-numbered rows and columns, following the same approach as with infrared datasets. Similarly to the infrared datasets, Visible Dataset 1 consisted of a static scene with no movement over time. Visible Dataset 2 featured a moving target, where only a person holding a resolution chart moved horizontally at a constant speed. Visible Dataset 3 was captured by a camera moving horizontally at a constant speed, resulting in the entire scene moving uniformly. The simulated noise added to the visible datasets was as follows.

Case 3: Gaussian noise with a mean of zero and variance of 0.005 was added to each frame of the image sequence in the visible datasets for different scenes.

The denoising results are shown in Figure 13 and

Table 4. Quantitative comparisons of all the comparative methods for visible datasets (Bold = Best PSNR/SSIM; Underline = Second-best PSNR/SSIM).

Datasets	Index	Noisy	VBM4D	RPCA	LRTDTV	LRTDGS	ASTTV-NTLA	SRSTT	bt-LPTVTD
Visible Dataset 1 (Static)	PSNR	23.0471	33.2106	30.9034	32.0976	31.8630	28.6229	25.6912	32.3759
	SSIM	0.5107	0.9061	0.9076	0.9065	0.8998	0.8151	0.7838	0.9147
Visible Dataset 2 (Moving Target)	PSNR	23.0788	31.8301	23.6049	27.0679	26.7372	27.0442	22.8561	30.3463
	SSIM	0.5384	0.8993	0.7728	0.8057	0.7969	0.7901	0.6888	0.8894
Visible Dataset 3 (Moving Camera)	PSNR	23.0618	29.9263	19.6028	24.9982	24.6559	26.1916	20.8633	28.4749
	SSIM	0.5149	0.8767	0.5144	0.7269	0.7032	0.7720	0.5407	0.8506

. In the noise suppression simulation experiments on the visible datasets, VBM4D performed the best, followed by bt-LPTVTD. The denoising effects of the various contrast algorithms were similar in static scenes. However, when moving objects appeared in a scene, only VBM4D and bt-LPTVTD maintained good denoising effects. Owing to the VBM4D block-matching and 4D filtering strategies, both the PSNR and SSIM achieved leading levels. Figure 15 shows the PSNR and SSIM performance of each method on the visible datasets. VBM4D performed the best, followed by bt-LPTVTD. It is worth noting that the calculations of the PSNR and SSIM were based on individual frames and were then averaged over multiple frames. Even when there was inter-frame variation, VBM4D could achieve a high PSNR and SSIM. Therefore, the best PSNR and SSIM of VBM4D do not necessarily indicate that its TRN denoising effect is the best. This will be verified in subsequent real experiments and can be observed in the slices.

Figure 14 shows the results of the real noise experiments on the visible datasets. The vertical slice at the yellow line is displayed in the yellow box on the right, clearly showing the denoising effect of TRN. It can be seen that the low-light image is severely contaminated by temporal Poisson noise. The noise level improved after denoising using different denoising methods. The LRTDTV, LRTDGS, ASTTV-NTLA, and SRSTT methods either failed to effectively suppress the noise (as shown in Figure 14a) or lost the details of the image (as shown in the red box in Figure 14b). RPCA performs well in denoising the static part but loses the information of the dynamic part. VBM4D performs well under weak noise conditions (as shown in Figure 14b) but can cause detail loss under strong noise conditions (as shown in Figure 15a), and the temporal noise removal is incomplete (as shown in the slice in the yellow box in Figure 14a). bt-LPTVTD has stable denoising effects under both strong and weak noise conditions, achieves denoising while preserving the details of the static and dynamic parts of the image, and has the best visual effect.

The robustness and generalizability of our bt-LPTVTD method were further evaluated by testing its performance on Poisson and mixed noise, noise types frequently encountered in low-light imaging. The results of these additional experiments are available at [https://zenodo.org/doi/10.5281/zenodo.10473803 (accessed on 22 March 2025)].

5.4. Ablation Study

To demonstrate the contribution of each component in bt-LPTVTD, an ablation study was conducted using a 30-frame sequence from Case 3 as an example. The key components, namely the bidirectional twisted tensor, total variation, and RPCA, were individually removed to assess their specific effects and benefits.

Table 5. Ablation experiments on bidirectional twisted tensors. Bold = Best result.

Datasets	Index	Noisy Image	With Bidirectional Twisted Tensor	Without Bidirectional Twisted Tensor
Visible Dataset 1 (Static)	PSNR	23.043	32.6948	32.3837
	SSIM	0.5132	0.9241	0.9216
	Running time (s)	/	69.6815	99.5749
Visible Dataset 2 (Moving Target)	PSNR	23.0728	29.5692	28.8052
	SSIM	0.5463	0.8857	0.8705
	Running time (s)	/	70.9823	94.9965
Visible Dataset 3 (Moving Camera)	PSNR	23.0702	28.926	28.0386
	SSIM	0.5109	0.8684	0.8456
	Running time (s)	/	69.9584	101.4699

presents the results of the ablation study on the bidirectional twisted tensor. The results demonstrate that the proposed method effectively enhances restoration performance. Across all three tested scenarios, a significant improvement in PSNR was observed, with an average gain of approximately 0.6 dB. Notably, the experimental results indicate a positive correlation between the proportion of motion regions in the datasets and the performance gain in both PSNR and SSIM, suggesting that a larger motion region proportion leads to more substantial improvements. Furthermore, the significant reduction in the size of each slice in the t-SVD within the proposed method resulted in a considerable improvement in the computational efficiency of the SVD process. The algorithm runtime for all three scenarios decreased from approximately 100 s to approximately 70 s, representing a reduction of approximately 30%, indicating a substantial improvement in algorithm efficiency.

Table 6. Ablation experiments on total variations. Bold = Best result.

Datasets	Index	Noisy Image	With Bidirectional Twisted Tensor	Without Bidirectional Twisted Tensor
Visible Dataset 1 (Static)	PSNR	23.043	32.6948	28.9195
	SSIM	0.5132	0.9241	0.7755
Visible Dataset 2 (Moving Target)	PSNR	23.0728	29.5692	27.5202
	SSIM	0.5463	0.8857	0.7683
Visible Dataset 3 (Moving Camera)	PSNR	23.0702	28.926	27.5034
	SSIM	0.5109	0.8684	0.7470

presents the results of the ablation study focusing on the impact of TV regularization. The findings unequivocally demonstrate that the integration of TV regularization significantly bolsters restoration performance, yielding a pronounced noise suppression effect. Marked improvements in both PSNR and SSIM metrics were observed across all tested scenarios. Specifically, in the static scenario, the PSNR value exhibited a substantial increase of 3.7753 dB, accompanied by a concomitant rise of 0.1486 in the SSIM. Similarly, in the moving-target scenario, a notable PSNR gain of 2.049 dB and an SSIM enhancement of 0.1174 were recorded. Finally, in the moving-camera scenario, the PSNR value increased by 1.4226 dB, with a corresponding SSIM improvement of 0.1214. These substantial performance gains underscore the inherent limitations of relying solely on the bt-LPTNN method for adaptability across diverse scenarios. The introduction of TV regularization demonstrably enhances the model’s adaptability and robustness, thereby leading to more effective TRN suppression.

Table 7. Ablation experiments on RPCA. Bold = Best result.

Datasets	Index	Noisy Image	With Bidirectional Twisted Tensor	Without Bidirectional Twisted Tensor
Visible Dataset 1 (Static)	PSNR	23.043	32.6948	29.4027
	SSIM	0.5132	0.9241	0.8487
Visible Dataset 2 (Moving Target)	PSNR	23.0728	29.5692	28.0195
	SSIM	0.5463	0.8857	0.8419
Visible Dataset 3 (Moving Camera)	PSNR	23.0702	28.926	28.8142
	SSIM	0.5109	0.8684	0.8654

presents the results of the ablation study on RPCA. The results demonstrate that employing RPCA to extract detailed information from the noise tensor effectively enhances the restoration of static components. The impact of RPCA becomes more significant with an increasing proportion of static temporal elements within the scene. Notably, the PSNR improved by 3.2921 dB, 1.5497 dB, and 0.1118 dB in the static, moving-target, and moving-camera scenarios, respectively, while the SSIM exhibited slight improvements across all scenarios. These gains highlight the effectiveness of RPCA in refining the restoration process, particularly in scenarios with more static content.

5.5. Running Time

All comparative algorithms were run on the same hardware platform, which used an Intel Core i5-8500 @ 3.00 GHz CPU and 16 GB of memory. The average processing time for each frame was obtained by calculating a tensor comprising a sequence of 30 continuous images, as listed in

Table 8. Running time (seconds).

Methods	VBM4D	RPCA	LRTDTV	LRTDGS	ASTTV-NTLA	SRSTT	bt-LPTVTD
Running time	6.7856	1.4288	11.1344	7.8910	11.8338	29.0100	2.0809

. Owing to the utilization of the bidirectional twisted tensor, the proposed bt-LPTVTD algorithm is significantly more efficient than the other tensor decomposition algorithms. Additionally, bt-LPTVTD can achieve parallel processing of consecutive tensor blocks by only requiring the collection of 30 continuous frames to form the input tensor without waiting for the computation of the previous tensor to complete. This effectively improved the efficiency of the algorithm in processing consecutive image sequences.

6. Discussion

This study introduces bt-LPTVTD, a novel TRN suppression method for uncooled infrared detectors, based on total variation regularization and low-rank tensor decomposition. This approach has demonstrated significant success in improving image quality, specifically enhancing the Signal-to-Noise Ratio (SNR) and Signal-to-Clutter Ratio (SCR) of processed images. Experimental results confirm that bt-LPTVTD effectively suppresses TRN while preserving essential image details, outperforming several state-of-the-art denoising algorithms across a range of challenging scenarios.

The core of bt-LPTVTD’s effectiveness lies in its combination of ASTTV regularization (for preserving spatial and temporal details), bt-LPTNN (for adaptive weighting and low-rank approximation), and RPCA (for recovering lost spatial information). This approach contrasts with methods that primarily target fixed-pattern noise or those that compromise motion information. While achieving strong denoising performance, bt-LPTVTD currently has a higher computational cost compared to some simpler methods. Therefore, a key focus of future work will be on optimizing computational efficiency through algorithmic improvements and hardware acceleration, enabling its deployment in real-time applications. The demonstrated robustness and detail preservation capabilities of bt-LPTVTD, however, make it a valuable preprocessing step for infrared small target recognition, where increased SNR and SCR are crucial. Beyond this specific application, the method holds promise for broader use in defense, medical imaging, and industrial inspection. Future research will also explore its application to other imaging modalities and the development of more comprehensive video quality metrics.

7. Conclusions

This paper proposes a bidirectional twisted tensor truncated nuclear norm (bt-LPTNN) based on the Laplace operator to effectively suppress TRN in low-signal-to-noise-ratio infrared and visible images. The bt-LPTNN can adaptively assign different weights to all singular values, and the bidirectional twisted tensor can better capture temporal information while preserving horizontal and vertical spatial domain information. In addition, the weighted spatiotemporal total variation regularization nonconvex tensor approximation method can fully utilize prior structural information and better preserve detailed textures. Furthermore, the use of RPCA on the noise tensor can effectively extract spatial domain information, resulting in a better performance of the algorithm in static scenes. These improvements constitute the proposed bt-LPTVTD method, whose main contributions include the following:

(1)

A bidirectional twisted tensor truncated nuclear norm based on the Laplacian operator combined with a weighted spatiotemporal total variation regularization nonconvex tensor approximation method is proposed. The bidirectional twisted tensor can better capture temporal information while preserving horizontal and vertical spatial information. The improved tensor recovery estimation method demonstrates more significant TRN suppression and more effectively preserves moving-target details.

(2)

To recover spatial information that may be lost during the tensor estimation process, RPCA is further utilized to extract spatial information from the noise tensor. As a result, the proposed method achieves improved detail recovery for the static components of the scene.

(3)

An augmented Lagrange multiplier algorithm is designed to solve the proposed bt-LPTVTD model.

Numerous experimental results demonstrate that, compared to existing methods that combine TV and tensor decomposition, the bt-LPTVTD method exhibits a superior TRN suppression effect while considering scene adaptability and robustness, not only providing better denoising performance but also showing strong adaptability to various noise types and scene conditions.

Future research will be directed toward developing a suitable method for FPN suppression during the tensor decomposition process, with the objective of achieving simultaneous suppression of both types of noise. In addition, denoising algorithms with high operational efficiencies will be developed for engineering applications, and further investigations will be conducted to develop simplification methods for the bt-LPTVTD model to improve its operational speed and expand its application scope to diverse scenarios. Future work also includes the development of more suitable video quality metrics that account for temporal information.