Optimizing Dynamic Mode Decomposition for Video Denoising via Plug-and-Play Alternating Direction Method of Multipliers ^†

Abstract

Dynamic mode decomposition (DMD) is a powerful tool for separating the background and foreground in videos. This algorithm decomposes a video into dynamic modes, called DMD modes, to facilitate the extraction of the near-zero mode, which represents the stationary background. Simultaneously, it captures the evolving motion in the remaining modes, which correspond to the moving foreground components. However, when applied to noisy video, this separation leads to degradation of the background and foreground components, primarily due to the noise-induced degradation of the DMD mode. This paper introduces a novel noise removal method for the DMD mode in noisy videos. Specifically, we formulate a minimization problem that reduces the noise in the DMD mode and the reconstructed video. The proposed problem is solved using an algorithm based on the plug-and-play alternating direction method of multipliers (PnP-ADMM). We applied the proposed method to several video datasets with different levels of artificially added Gaussian noise in the experiment. Our method consistently yielded superior results in quantitative evaluations using peak-signal-to-noise ratio (PSNR) and structural similarity (SSIM) compared to naive noise removal methods. In addition, qualitative comparisons confirmed that our method can restore higher-quality videos than the naive methods.

1. Introduction

1.1. Background

Video processing is critical in surveillance and in-vehicle systems and specifically includes essential tasks such as noise removal, foreground/background separation, and object detection. The foreground/background separation process helps detect, identify, track, and recognize objects within a video sequence.

Dynamic mode decomposition (DMD) is useful for separating background and foreground components in various applications [1,2,3,4,5,6,7,8,9]. Initially applied in fluid dynamics, DMD has evolved into a powerful tool for analyzing the dynamics of nonlinear systems, as shown in research such as [10,11,12]. In background/foreground separation, the DMD method identifies a static background by performing a spatiotemporal decomposition of video frames. It effectively distinguishes between static modes and the remaining dynamic modes, separating a static background from a dynamic foreground.

In low-light conditions and using high-sensitivity settings, the captured video exhibits noticeable noise levels due to amplified sensor noise. This amplification occurs because camera sensors operating at high sensitivity are more susceptible to capturing and amplifying random electrical signals. However, sensor noise often deteriorates the DMD mode when attempting to separate foreground and background components from noisy videos using the DMD algorithm. This issue has also been mentioned in the field of fluid analysis, where sensor noise can introduce bias errors and reduce the accuracy of the analysis of fluids.

To address this limitation, some researchers have proposed the total-least-squares DMD (tlsDMD) algorithm to mitigate bias errors due to sensor noise [13,14]. However, tlsDMD-based methods are ineffective at removing spatial noise to separate a noisy video into foreground and background because they lack prior knowledge that promotes image smoothness.

1.2. Related Work

In [10], Schmid introduced the basic DMD algorithm and explained its relevance to standard methods used in fluid analysis for atmospheric or oceanographic data. The potential of the DMD algorithm was demonstrated through several scenarios, including a plane channel flow, flow over a two-dimensional cavity, wake flow behind a flexible membrane, and a jet passing between two cylinders. The demonstrations showed the ability of this algorithm to analyze fluid flows and identify critical physical mechanisms that govern them, highlighting its power and versatility.

In [13,14], the vulnerability of the DMD algorithm to sensor noise was mentioned. The basic DMD algorithm does not take sensor noise into account. When decomposing snapshots degraded by sensor noise, the estimated eigenvalues deviate from the ideal values due to noise bias. Hemati et al. proposed the tlsDMD algorithm, which estimates the bias due to sensor noise in forward and backward DMD mode estimation of snapshots [14]. This algorithm calculates DMD modes and their eigenvalues while excluding the estimated bias. As a result, the bias caused by sensor noise can be removed, and simulation experiments showed that the estimated eigenvalues are close to those calculated for snapshots without sensor noise. Dawson et al. analytically derived a formula that explicitly shows how DMD is affected by noise, assuming that sensor noise is uncorrelated with system dynamics [13]. They complemented the derivation of the tlsDMD algorithm. However, this algorithm aims to remove the sensor noise bias, and the noise removal accuracy of the DMD mode is insufficient. In addition, they do not consider the reconstruction error of snapshots. Therefore, when applied to videos for foreground/background separation applications, it is impossible to remove sensor noise from reconstructed video frames sufficiently and their DMD modes.

Various noise removal methods have been proposed for images and videos, including optimization-based approaches such as total variation (TV) regularization [15,16,17,18,19,20,21] and filter-based noise removal methods such as block matching 3D (BM3D) [22,23,24,25]. The TV is designed to represent the total magnitude of the vertical and horizontal discrete gradients of an image and promotes the local smoothness property in optimization [15,16,17,18,19]. In the case of noise removal, the TV effectively reduces noise by emphasizing spatial smoothness while preserving edges and structures. Although BM3D was proposed over a decade ago, it remains one of the most advanced methods for denoising images and videos [22,23]. It works by partitioning the image into blocks, searching for similar blocks, and then thresholding their noise in the 3D-transformed domain using the discrete cosine transform (DCT). BM3D uses local and non-local similarities in the image to effectively reduce noise while preserving image structure and texture.

However, these noise removal methods do not explicitly account for the spatial smoothness and texture of the DMD mode. Improving the spatial smoothness and texture of both the video frames and their DMD modes is critical to obtaining reliable results for practical video analysis in the presence of noise. Additionally, for foreground/background separation applications, noise removal on the DMD mode obtained by its decomposition must be considered.

1.3. Contribution

This paper introduces a novel noise removal method for the DMD mode obtained by applying DMD to noisy videos. We formulate a minimization problem within the plug-and-play framework that aims to simultaneously reduce the noise in DMD modes and its reconstructed videos. To solve the proposed problem, we introduce an algorithm based on the plug-and-play alternating direction method of multipliers (PnP-ADMM). The experimental results demonstrate the effectiveness of the proposed method by comparing it with naive noise removal methods. The main contributions of this paper are as follows:

• Introducing a novel minimization problem that simultaneously removes noise from DMD modes and improves their reconstructed video quality. This problem includes two implicit regularization terms for the DMD modes and their reconstructed video, along with two constraints on the reconstructed video: one for reconstruction error and the other to ensure real numbers.
• The development of the PnP-ADMM algorithm is based on the plug-and-play framework and Gaussian denoisers. This algorithm solves the proposed minimization problem and aims to obtain optimal DMD modes capable of reconstructing a smooth and noiseless video.
• Two advanced noise removal methods, the total variation (TV) algorithm and BM3D, are employed as Gaussian denoisers to implicitly regularize the DMD modes and their reconstructed video within the optimization algorithm.

In the previous study [26], we used the TV denoiser with the PnP-ADMM algorithm to remove noise from the DMD modes obtained by decomposing the observed noisy videos. Since the DMD modes are complex numbers, the reconstructed video may have values in the imaginary part. Although the reconstructed video must contain real numbers, such constraints were not explicitly considered when formulating the optimization problem. In the proposed method, we replaced the TV denoiser with the BM3D denoiser to improve the noise removal performance. Additionally, we added a constraint to restrict the reconstructed video obtained by optimal DMD mode to real numbers in the optimization problem.

The remainder of this paper is organized as follows. In Section 2, we present mathematical preliminaries, a DMD algorithm, a PnP-ADMM algorithm, some proximal tools, and total variation regularization. Section 3 introduces the proposed minimization problem for noise removal of the DMD mode. In Section 4, several examples are presented and compared with some naive noise removal methods to verify the effectiveness of the proposed method. Finally, Section 5 concludes the paper.

2. Preliminaries

Throughout this paper, bold-faced lowercase and uppercase letters indicate vectors and matrices, respectively. The notations ${\mathbb{R}}^N$ and ${\mathbb{C}}^N$ denote real- and complex-valued vector spaces of N dimensions, respectively. We define the notations ${\mathbb{R}}^{N\times M}$ and ${\mathbb{C}}^{N\times M}$ as the set of $N\times M$ real-valued and complex-valued matrices, respectively. The symbols ${(·)}^{\top }$ and ${(·)}^{*}$ denote the operations of non-conjugate and conjugate transpose of vectors and matrices, respectively. The symbol diag(X) denotes the operation of extracting the diagonal components of a diagonal matrix X and converting it into a column vector.

2.1. Dynamic Mode Decomposition

The DMD algorithm is defined for pairs of N-dimentional data $\{{\mathbf{x}}_i,{\mathbf{y}}_i\}$ satisfying ${\mathbf{y}}_i=\mathbf{A}{\mathbf{x}}_i$$(i=1,\dots ,M)$, for some matrix $\mathbf{A}\in {\mathbb{R}}^{N\times N}$. These vectors are sampled by equispaced snapshots of a dynamical system. However, the matrix $\mathbf{A}$ is not completely determined by the snapshots. The DMD algorithm estimates $\mathbf{A}$ such that satisfying $\mathbf{Y}\approx \mathbf{AX}$, where $\mathbf{Y}:=[{\mathbf{y}}_1,\dots ,{\mathbf{y}}_M]$ and $\mathbf{X}:=[{\mathbf{x}}_1,\dots ,{\mathbf{x}}_M]$. Several methods have been proposed to compute DMD [10,13,14,27,28].

In this paper, we use the basic DMD algorithm [10] described as follows:

(i)

Calculate the (reduced) singular value decomposition (SVD) of the matrix $\mathbf{X}$ as $\mathbf{X}=\mathbf{U}\mathbf{S}{\mathbf{V}}^{*}$, where $\mathbf{U}\in {\mathbb{C}}^{N\times r}$, $\mathbf{S}\in {\mathbb{C}}^{r\times r}$, and $\mathbf{V}\in {\mathbb{C}}^{M\times r}$, with the rank r.

(ii)

Let $\tilde{\mathbf{A}}$ be defined by $\tilde{\mathbf{A}}={\mathbf{U}}^{*}\mathbf{Y}\mathbf{V}{\mathbf{S}}^{-1}$.

(iii)

Compute the eigenvalue decomposition of $\tilde{\mathbf{A}}$ as $\tilde{\mathbf{A}}\mathbf{W}=\mathbf{W}\mathsf{\Lambda }$, where $\mathbf{W}:=[{\mathbf{w}}_1,\dots ,{\mathbf{w}}_r]$ is a matrix configured by arranging the eigenvectors ${\mathbf{w}}_i\in {\mathbb{C}}^r(i=1,\dots ,r)$ and $\mathsf{\Lambda }$ is a diagonal matrix having eigenvalues ${\lambda }_i(i=1,\dots ,r)$ as the diagonal elements.

(iv)

The DMD mode $\mathsf{\Phi }:=[{\mathsf{\varphi }}_1,\dots ,{\mathsf{\varphi }}_r]({\mathsf{\varphi }}_i\in {\mathbb{C}}^N)$ is obtained by $\mathsf{\Phi }=\mathbf{U}\mathbf{W}$.

(v)

Then, we define $\mathsf{\Sigma }\in {\mathbb{C}}^{r\times M}$ as

$$\mathsf{\Sigma }:=[\mathrm{diag}\left({\mathsf{\Lambda }}^0\right)\mathrm{diag}\left({\mathsf{\Lambda }}^1\right)\cdots \mathrm{diag}\left({\mathsf{\Lambda }}^{M-1}\right)].$$

(1)

(vi)

Estimate the diagonal matrix $\mathbf{B}\in {\mathbb{C}}^{r\times r}$ by minimizing the cost function

$$E\left(\mathbf{B}\right):={\parallel \mathbf{X}-\mathsf{\Phi }\mathbf{B}\mathsf{\Sigma }\parallel }_F^2.$$

(2)

(vii)

Finally, $\mathbf{X}$ is represented by $\mathsf{\Phi }\mathbf{B}\mathsf{\Sigma }$ as

$$\mathbf{X}\approx \mathsf{\Phi }\mathbf{B}\mathsf{\Sigma }.$$

(3)

In this manner, the DMD algorithm decomposes $\mathbf{X}$ into $\mathsf{\Phi },\mathbf{B},$ and $\mathsf{\Sigma }$, where $\mathsf{\Phi }$ is the set of dynamic modes of observed dynamical systems, each diagonal element of $\mathbf{B}$ is the amplitude of each mode, and each row of $\mathsf{\Sigma }$ is a Vandermonde matrix describing the temporal evolution of each mode.

2.2. Plug-and-Play Alternating Direction Method of Multipliers

The alternating direction method of multipliers (ADMM) [29] is a proximal splitting algorithm for convex optimization problems of the form

$$\underset{\mathbf{x}\in {\mathbb{R}}^{N_1},\mathbf{z}\in {\mathbb{R}}^{N_2}}{min}F\left(\mathbf{x}\right)+G\left(\mathbf{z}\right)\mathrm{s}.\mathrm{t}.\mathbf{z}=\mathbf{Lx},$$

(4)

where F and G are usually assumed to be a quadratic and proximable function, respectively, and $\mathbf{L}\in {\mathbb{R}}^{N_2\times N_1}$ is a matrix with full-column rank. For any ${\mathbf{x}}^{\left(0\right)}\in {\mathbb{R}}^{N_1},{\mathbf{z}}^{\left(0\right)}\in {\mathbb{R}}^{N_2},{\mathbf{b}}^{\left(0\right)}\in {\mathbb{R}}^{N_2}$ and $\rho >0$, the ADMM algorithm is given by

$$⌊\begin{array}{cc}\hfill {\mathbf{x}}^{(t+1)}& =arg\underset{\mathbf{x}}{min}\left\{F\left(\mathbf{x}\right)+\frac{\rho }{2}{\parallel {\mathbf{z}}^{\left(t\right)}-\mathbf{Lx}-{\mathbf{b}}^{\left(t\right)}\parallel }_2^2\right\},\hfill \\ \hfill {\mathbf{z}}^{(t+1)}& =arg\underset{\mathbf{z}}{min}\left\{G\left(\mathbf{z}\right)+\frac{\rho }{2}{\parallel \mathbf{z}-{\mathbf{Lx}}^{(t+1)}-{\mathbf{b}}^{\left(t\right)}\parallel }_2^2\right\},\hfill \\ \hfill {\mathbf{b}}^{(t+1)}& ={\mathbf{b}}^{\left(t\right)}+{\mathbf{Lx}}^{(t+1)}-{\mathbf{z}}^{(t+1)},\hfill \end{array}$$

(5)

where the superscript $\left(t\right)$ denotes the iteration number. The sequence generated by Equation (5) converges quickly to an optimal solution of Equation (4).

In PnP-ADMM [30,31], the solution of the sub-problem with respect to $\mathbf{z}$ (assuming $\mathbf{L}$ is the identity matrix) is replaced by an off-the-shelf noise removal algorithm, to yield

$${\mathbf{z}}^{(t+1)}={\mathcal{D}}_{\sigma }\left({\mathbf{x}}^{(t+1)}+{\mathbf{b}}^{\left(t\right)}\right),$$

(6)

where ${\mathcal{D}}_{\sigma }$ denotes the Gaussian denoiser and $\sigma$ is the standard deviation of the assumed additive white Gaussian noise (AWGN).

2.3. Proximal Tools

The proximity operator [32] is a key tool of proximal splitting techniques. Let $\mathbf{x}\in {\mathbb{R}}^N$ be an input vector. For any $\gamma >0$, the proximity operator of f over ${\mathbb{R}}^N$ is defined by

$${\mathrm{prox}}_{\gamma f}\left(\mathbf{x}\right):=arg\underset{\mathbf{y}\in {\mathbb{R}}^N}{min}f\left(\mathbf{y}\right)+\frac{1}{2\gamma }{\parallel \mathbf{x}-\mathbf{y}\parallel }^2.$$

(7)

For a given nonempty closed convex set $\mathcal{C}$, the indicator function of $\mathcal{C}$ is defined by

$${\iota }_{\mathcal{C}}\left(\mathbf{x}\right):=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}\mathbf{x}\in \mathcal{C},\hfill \\ +\infty ,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(8)

The proximity operator of ${\iota }_{\mathcal{C}}$ is expressed as

$${\mathrm{prox}}_{\gamma {\iota }_{\mathcal{C}}}\left(\mathbf{x}\right):=arg\underset{\mathbf{y}\in {\mathbb{R}}^N}{min}{\iota }_{\mathcal{C}}\left(\mathbf{x}\right)+\frac{1}{2\gamma }{\parallel \mathbf{x}-\mathbf{y}\parallel }_2^2.$$

(9)

The solution of ${\mathrm{prox}}_{\gamma {\iota }_{\mathcal{C}}}$ should be in the set $\mathcal{C}$ and minimize ${\parallel \mathbf{x}-\mathbf{y}\parallel }_2^2$. Thus, for any index $\gamma >0$, this proximity operator is equivalent to the metric projection onto $\mathcal{C}$, i.e., ${\mathcal{P}}_{\mathcal{C}}\left(\mathbf{x}\right)={\mathrm{prox}}_{\gamma {\iota }_{\mathcal{C}}}\left(\mathbf{x}\right)$.

Let $\mathbf{l}$ and $\mathbf{u}$$\in {\mathbb{R}}^N$ be the lower and upper bounds, respectively. The box constraint forces each element of $\mathbf{x}$ into the dynamic range $[l_i,u_i]$ for $i=1,\dots ,N$, and its closed convex set is defined as

$${\mathcal{C}}_{[\mathbf{l},\mathbf{u}]}:=\left\{\mathbf{x}\in {\mathbb{R}}^N\mid l_i\le x_i\le u_i(i=1,\dots ,N)\right\}.$$

(10)

The computation of the metric projection onto ${\mathcal{C}}_{[\mathbf{l},\mathbf{u}]}$ for $i=1,\dots ,N$ is given by

$${\left[{\mathcal{P}}_{{\mathcal{C}}_{[\mathbf{l},\mathbf{u}]}}\left(\mathbf{x}\right)\right]}_i=\left\{\begin{array}{cc}{l}_i,\hfill & \mathrm{if} x_i<l_i,\hfill \\ u_i,\hfill & \mathrm{if} x_i>u_i,\hfill \\ x_i,\hfill & \mathrm{if} l_i\le x_i\le u_i.\hfill \end{array}\right.$$

(11)

The ${\ell }_2$ ball constraint forces the Euclidean distance between a vector $\mathbf{x}$ and a centered vector $\mathbf{v}$ to be less than a radius $ϵ$, and its closed convex set is defined as

$${\mathcal{B}}_{\mathbf{v},ϵ}^2:=\left\{\mathbf{x}\in {\mathbb{R}}^N\mid {\parallel \mathbf{x}-\mathbf{v}\parallel }_2\le ϵ\right\}.$$

(12)

The computation of the metric projection onto ${\mathcal{B}}_{\mathbf{v},ϵ}^2$ is given by

$${\mathcal{P}}_{{\mathcal{B}}_{\mathbf{v},ϵ}^2}\left(\mathbf{x}\right)=\left\{\begin{array}{cc}\mathbf{x},\hfill & {\mathrm{if}\parallel \mathbf{x}-\mathbf{v}\parallel }_2\le ϵ,\hfill \\ \mathbf{v}+ϵ\frac{\mathbf{x}-\mathbf{v}}{{\parallel \mathbf{x}-\mathbf{v}\parallel }_2},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(13)

2.4. Total Variation

The total variation (TV) is defined as the total magnitude of the vertical and horizontal discrete gradients of an image [16]. When we utilize the TV as a regularization on minimization problems for images, it promotes the local smoothness of images to be estimated.

Let $\mathbf{x}\in {\mathbb{R}}^N$ be a vectorized grayscale image, where N is the total number of pixels. Also, let ${\mathbf{D}}_v$ and ${\mathbf{D}}_h\in {\mathbb{R}}^{N\times N}$ be the vertical and horizontal first-order differential operators with a Neumann boundary, respectively. Then, the differential operator with respect to $\mathbf{x}$ is defined by $\mathbf{D}:={\left[{\mathbf{D}}_v^{\top }{\mathbf{D}}_h^{\top }\right]}^{\top }\left(\in {\mathbb{R}}^{2N\times N}\right)$, and thus the TV is defined as [16,33,34]

$${\parallel \mathbf{x}\parallel }_{\mathrm{TV}}:={\parallel \mathbf{D}\mathbf{x}\parallel }_{1,2}=\sum _{i=1}^N\sqrt{{\left({\mathbf{D}}_v\mathbf{x}\right)}_i^2+{\left({\mathbf{D}}_h\mathbf{x}\right)}_i^2},$$

(14)

where ${\left({\mathbf{D}}_v\mathbf{x}\right)}_i$ and ${\left({\mathbf{D}}_h\mathbf{x}\right)}_i$ are the i-th element of ${\mathbf{D}}_v\mathbf{x}$ and ${\mathbf{D}}_h\mathbf{x}$, respectively.

The minimization problem with TV regularization, which is often used in PnP-ADMM as a denoiser, is defined as

$$\begin{array}{cc}\hfill {\mathbf{x}}^{★}& =arg\underset{\mathbf{x}\in {\mathbb{R}}^N}{min}{\lambda \parallel \mathbf{x}\parallel }_{\mathrm{TV}}+\frac{1}{2}{\parallel \mathbf{x}-{\mathbf{x}}_{\mathrm{in}}\parallel }_2^2,\hfill \end{array}$$

(15)

where ${\mathbf{x}}_{\mathrm{in}}$ is a vectorized input image and $\lambda >0$ is a balancing weight of two terms. We can find the optimal solution of Equation (15) by using the ADMM algorithm.

By introducing auxiliary variables $\mathbf{z}\in {\mathbb{R}}^{2N}$, we rewrite Equation (15) into the following equivalent expression:

$$\underset{\mathbf{x},\mathbf{z}}{min}{\lambda \parallel \mathbf{z}\parallel }_{\mathrm{TV}}+\frac{1}{2}{\parallel \mathbf{x}-{\mathbf{x}}_{\mathrm{in}}\parallel }_2^2\mathrm{s}.\mathrm{t}.\mathbf{z}=\mathbf{x}.$$

(16)

The algorithm for solving Equation (16) with $\rho >0$ is summarized in Algorithm 1. The update of $\mathbf{x}$ can be achieved by solving a simple quadratic minimization problem. The solution of the sub-problem with respect to $\mathbf{z}$ can be obtained for each sub-vector ${\mathbf{z}}^{{\mathcal{G}}_1},\dots ,{\mathbf{z}}^{{\mathcal{G}}_N}$, by

$${\mathbf{z}}^{{\mathcal{G}}_i}={\mathrm{prox}}_{{\lambda /\rho \parallel ·\parallel }_{\mathrm{TV}}}\left({\mathbf{y}}^{{\mathcal{G}}_i}\right)=max\left\{1-\frac{\lambda }{\rho }{\left(y_i^2+y_{i+N}^2\right)}^{-\frac{1}{2}},0\right\}{\mathbf{y}}^{{\mathcal{G}}_i},$$

(17)

where ${\mathbf{z}}^{{\mathcal{G}}_i}=:\{z_i,z_{i+N}\}$ and ${\mathbf{y}}^{{\mathcal{G}}_i}=:\{y_i,y_{i+N}\}$ are the i-th sub-vector of $\mathbf{z}$ and $\mathbf{y}$, respectively.

Algorithm 1 Solved algorithm for Equation (16)

1: Input: ${\mathbf{x}}_{\mathrm{in}},$ ${\mathbf{z}}^{\left(0\right)},$ ${\mathbf{b}}^{\left(0\right)}$, $\rho$, $\lambda$ 2: Output: ${\mathbf{x}}^{\left(t\right)}$ 3: while A stopping criterion is not satisfied do 4:    ${\mathbf{x}}^{(t+1)}\leftarrow arg{min}_{\mathbf{x}}\frac{1}{2}\parallel \mathbf{x}-{\mathbf{x}}_{\mathrm{in}}{\parallel }_2^2+\frac{\rho }{2}{\parallel {\mathbf{z}}^{\left(t\right)}-\mathbf{D}\mathbf{x}-{\mathbf{b}}^{\left(t\right)}\parallel }_2^2$ ; 5:    ${\mathbf{z}}^{(t+1)}\leftarrow {\mathrm{prox}}_{{\lambda /\rho \parallel ·\parallel }_{\mathrm{TV}}}\left(\mathbf{D}{\mathbf{x}}^{(t+1)}+{\mathbf{b}}^{\left(t\right)}\right)$ ; 6:    ${\mathbf{b}}^{(t+1)}\leftarrow {\mathbf{b}}^{\left(t\right)}+\mathbf{D}{\mathbf{x}}^{(t+1)}-{\mathbf{z}}^{(t+1)}$ ; 7:    $t\leftarrow t+1$; 8: end while

3. Proposed Methods

3.1. Data Model

We consider the following observation model

$${\mathbf{y}}_m={\mathbf{x}}_m+{\mathbf{n}}_m,$$

(18)

where ${\mathbf{x}}_m\in {\mathbb{R}}^N(m=1,\dots ,M+1)$ denotes a vectorized latent video frame, N is the number of pixels, $M+1$ is the number of frames, ${\mathbf{n}}_m\in {\mathbb{R}}^N$ is an AWGN vector, and ${\mathbf{y}}_m\in {\mathbb{R}}^N(m=1,\dots ,M+1)$ is a vectorized observed video frame. Furthermore, we defined the matrix form of $m=1,\dots ,M$ frames of the observed and decomposed video by using the above ${\mathbf{y}}_m$ and the DMD algorithm described in Section 2.1 as

$$\mathbf{Y}:=[{\mathbf{y}}_1{\mathbf{y}}_2\dots {\mathbf{y}}_M]\approx \widehat{\mathsf{\Phi }}\mathbf{B}\mathsf{\Sigma },$$

(19)

where $\widehat{\mathsf{\Phi }}\in {\mathbb{C}}^{N\times r}$ is the matrix consisting of noisy DMD modes arranged to the row direction. We assumed that DMD modes are degraded, while its amplitudes $\mathbf{B}$ and the temporal evolution $\mathsf{\Sigma }$ are scarcely affected by noise.

3.2. Minimization Problem

Our aim is to find a noiseless DMD mode matrix ${\mathsf{\Phi }}^{★}$ from a noisy observed video $\mathbf{Y}\approx \widehat{\mathsf{\Phi }}\mathbf{B}\mathsf{\Sigma }$. To estimate ${\mathsf{\Phi }}^{★}$, we formulate the following minimization problem:

$$\underset{\mathsf{\Phi }}{min}\alpha {\mathcal{R}}_r\left(\mathsf{\Phi }\mathbf{B}\mathsf{\Sigma }\right)+(1-\alpha ){\mathcal{R}}_m\left(\mathsf{\Phi }\right)\mathrm{s}.\mathrm{t}. {\parallel \mathbf{Y}-\mathsf{\Phi }\mathbf{B}\mathsf{\Sigma }\parallel }_F\le ϵ,\mathsf{\Phi }\mathbf{B}\mathsf{\Sigma }\in {\mathbb{R}}^{N\times M},$$

(20)

where ${\mathcal{R}}_r$ and ${\mathcal{R}}_m$ are regularization terms for a reconstructed video $\mathsf{\Phi }\mathbf{B}\mathsf{\Sigma }$ and a DMD mode matrix $\mathsf{\Phi }$, respectively, and $\alpha \in [0,1]$ is the balancing weight of these terms. The observed video matrix consists of real numbers, but the reconstructed video may contain complex numbers due to the nature of the matrices obtained by the DMD algorithm, which are complex. Therefore, we introduce a real-valued constraint on the reconstructed video in the minimization problem.

To find a solution of Equation (20), we employ the PnP-ADMM algorithm described in Section 2.2.

3.3. Optimization

The minimization problem Equation (20) is not directly applicable to PnP-ADMM. We reformulate it in a form that can be applied to PnP-ADMM. First, we define the convex set ${\mathcal{B}}_{\mathbf{Y},ϵ}^F$ as

$${\mathcal{B}}_{\mathbf{Y},ϵ}^F:=\left\{\mathbf{X}\in {\mathbb{R}}^{N\times M}\mid {\parallel \mathbf{X}-\mathbf{Y}\parallel }_F\le ϵ\right\}.$$

(21)

Then, we reformulate Equation (20) into the following unconstrained problem:

$$\underset{\mathsf{\Phi }}{min}\alpha {\mathcal{R}}_r\left(\mathsf{\Phi }\mathbf{B}\mathsf{\Sigma }\right)+(1-\alpha ){\mathcal{R}}_m\left(\mathsf{\Phi }\right)+{\iota }_{{\mathcal{B}}_{\mathbf{Y},ϵ}^F}\left(\mathsf{\Phi }\mathbf{B}\mathsf{\Sigma }\right)+{\iota }_{{\mathbb{R}}^{N\times M}}\left(\mathsf{\Phi }\mathbf{B}\mathsf{\Sigma }\right),$$

(22)

where ${\iota }_{{\mathcal{B}}_{\mathbf{Y},ϵ}^F}(·)$ is the indicator function of ${\mathcal{B}}_{\mathbf{Y},ϵ}^F$. This function guarantees that the Frobenius norm of $\mathbf{Y}-\mathsf{\Phi }\mathbf{B}\mathsf{\Sigma }$ is less than or equal to $ϵ$. Similarly, ${\iota }_{{\mathbb{R}}^{N\times M}}(·)$ is the indicator function of ${\mathbb{R}}^{N\times M}$. This function guarantees that the reconstructed video is composed of real numbers. Thus, the role of the third and fourth terms of Equation (22) correspond to the constraints of the minimization problem Equation (20). Furthermore, by introducing auxiliary variables ${\mathbf{Z}}_1\in {\mathbb{C}}^{N\times M},{\mathbf{Z}}_2\in {\mathbb{C}}^{N\times r},{\mathbf{Z}}_3\in {\mathbb{C}}^{N\times M},$ and ${\mathbf{Z}}_4\in {\mathbb{C}}^{N\times M}$, we rewrite the minimization problem Equation (22) into the following equivalent expression:

$$\underset{\mathsf{\Phi },{\mathbf{Z}}_i(i=1,2,3,4)}{min}\alpha {\mathcal{R}}_r\left({\mathbf{Z}}_1\right)+(1-\alpha ){\mathcal{R}}_m\left({\mathbf{Z}}_2\right)+{\iota }_{{\mathcal{B}}_{\mathbf{Y},ϵ}^F}\left({\mathbf{Z}}_3\right)+{\iota }_{{\mathbb{R}}^{N\times M}}\left({\mathbf{Z}}_4\right),\mathrm{s}.\mathrm{t}.{\mathbf{Z}}_1=\mathsf{\Phi }\mathbf{B}\mathsf{\Sigma },{\mathbf{Z}}_2=\mathsf{\Phi },{\mathbf{Z}}_3=\mathsf{\Phi }\mathbf{B}\mathsf{\Sigma },{\mathbf{Z}}_4=\mathsf{\Phi }\mathbf{B}\mathsf{\Sigma }.$$

(23)

The minimization problem Equation (23) can be applied to PnP-ADMM. The process of PnP-ADMM for solving Equation (23) with ${\rho }_i(i=1,2,3,4)$ is summarized in Algorithm 2.

Algorithm 2 Proposed algorithm for Equation (23)

1: Input: $\mathbf{Y},$ ${\mathbf{Z}}_i^{\left(0\right)}$, ${\mathsf{\Theta }}_i^{\left(0\right)},{\rho }_i$ $(i=1,2,3,4)$, $\alpha$, $ϵ$ 2: Output: ${\mathsf{\Phi }}^{\left(t\right)}$ 3: while A stopping criterion is not satisfied do 4:    ${\mathsf{\Phi }}^{\left(t+1\right)}\leftarrow arg{min}_{\mathsf{\Phi }}\frac{{\rho }_1}{2}\parallel {\mathbf{Z}}_1^{\left(t\right)}-\mathsf{\Phi }\mathbf{B}\mathsf{\Sigma }-{\mathsf{\Theta }}_1^{\left(t\right)}{\parallel }_F^2+\frac{{\rho }_2}{2}\parallel {\mathbf{Z}}_2^{\left(t\right)}-\mathsf{\Phi }-{\mathsf{\Theta }}_2^{\left(t\right)}{\parallel }_F^2+\frac{{\rho }_3}{2}\parallel {\mathbf{Z}}_3^{\left(t\right)}-\mathsf{\Phi }\mathbf{B}\mathsf{\Sigma }-{\mathsf{\Theta }}_3^{\left(t\right)}{\parallel }_F^2+\frac{{\rho }_4}{2}{\parallel {\mathbf{Z}}_4^{\left(t\right)}-\mathsf{\Phi }\mathbf{B}\mathsf{\Sigma }-{\mathsf{\Theta }}_4^{\left(t\right)}\parallel }_F^2$; 5:    ${\mathbf{Z}}_1^{(t+1)}\leftarrow {\mathcal{D}}_{{\mathcal{R}}_{r,\alpha /{\rho }_1}}\left({\mathsf{\Phi }}^{(t+1)}\mathbf{B}\mathsf{\Sigma }+{\mathsf{\Theta }}_1^{\left(t\right)}\right)$; 6:    ${\mathbf{Z}}_2^{(t+1)}\leftarrow {\mathcal{D}}_{{\mathcal{R}}_{m,(1-\alpha )/{\rho }_2}}\left({\mathsf{\Phi }}^{(t+1)}+{\mathsf{\Theta }}_2^{\left(t\right)}\right)$; 7:    ${\mathbf{Z}}_3^{(t+1)}\leftarrow {\mathcal{P}}_{{\mathcal{B}}_{\mathbf{Y},ϵ}^F}\left({\mathsf{\Phi }}^{(t+1)}\mathbf{B}\mathsf{\Sigma }+{\mathsf{\Theta }}_3^{\left(t\right)}\right)$; 8:    ${\mathbf{Z}}_4^{(t+1)}\leftarrow {\mathcal{P}}_{{\mathbb{R}}^{N\times M}}\left({\mathsf{\Phi }}^{(t+1)}\mathbf{B}\mathsf{\Sigma }+{\mathsf{\Theta }}_4^{\left(t\right)}\right)$; 9:    ${\mathsf{\Theta }}_1^{(t+1)}\leftarrow {\mathsf{\Theta }}_1^{\left(t\right)}+{\mathsf{\Phi }}^{(t+1)}\mathbf{B}\mathsf{\Sigma }-{\mathbf{Z}}_1^{(t+1)}$; 10:    ${\mathsf{\Theta }}_2^{(t+1)}\leftarrow {\mathsf{\Theta }}_2^{\left(t\right)}+{\mathsf{\Phi }}^{(t+1)}-{\mathbf{Z}}_2^{(t+1)}$; 11:    ${\mathsf{\Theta }}_3^{(t+1)}\leftarrow {\mathsf{\Theta }}_3^{\left(t\right)}+{\mathsf{\Phi }}^{(t+1)}\mathbf{B}\mathsf{\Sigma }-{\mathbf{Z}}_3^{(t+1)}$; 12:    ${\mathsf{\Theta }}_4^{(t+1)}\leftarrow {\mathsf{\Theta }}_4^{\left(t\right)}+{\mathsf{\Phi }}^{(t+1)}\mathbf{B}\mathsf{\Sigma }-{\mathbf{Z}}_4^{(t+1)}$; 13:    $t\leftarrow t+1$; 14: end while

The update of $\mathsf{\Phi }$ in step 4 of Algorithm 2 is achieved by solving the quadratic minimization problem. The optimal solution satisfies the condition that the partial derivative of the following quadratic cost function with respect to $\mathsf{\Phi }$ is zero (hereafter, the superscript $\left(t\right)$ is omitted for simplicity):

$$E\left(\mathsf{\Phi }\right)=\frac{{\rho }_1}{2}\parallel {\mathbf{Z}}_1-\mathsf{\Phi }\mathbf{B}\mathsf{\Sigma }-{\mathsf{\Theta }}_1{\parallel }_F^2+\frac{{\rho }_2}{2}\parallel {\mathbf{Z}}_2-\mathsf{\Phi }-{\mathsf{\Theta }}_2{\parallel }_F^2+\frac{{\rho }_3}{2}\parallel {\mathbf{Z}}_3-\mathsf{\Phi }\mathbf{B}\mathsf{\Sigma }-{\mathsf{\Theta }}_3{\parallel }_F^2+\frac{{\rho }_4}{2}{\parallel {\mathbf{Z}}_4-\mathsf{\Phi }\mathbf{B}\mathsf{\Sigma }-{\mathsf{\Theta }}_4\parallel }_F^2.$$

(24)

By setting the first-order derivative to zero, the optimal solution is determined by solving the system of linear equations:

$$\begin{array}{cc}\hfill & \mathsf{\Phi }\mathsf{\Psi }=\mathsf{\Xi },\hfill \\ \hfill & \left\{\begin{array}{c}\mathsf{\Psi }={\rho }_1\mathbf{B}\mathsf{\Sigma }{\mathsf{\Sigma }}^{*}{\mathbf{B}}^{*}+{\rho }_2\mathbf{I}+{\rho }_3\mathbf{B}\mathsf{\Sigma }{\mathsf{\Sigma }}^{*}{\mathbf{B}}^{*}+{\rho }_4\mathbf{B}\mathsf{\Sigma }{\mathsf{\Sigma }}^{*}{\mathbf{B}}^{*},\hfill \\ \mathsf{\Xi }={\rho }_2({\mathbf{Z}}_1-{\mathsf{\Theta }}_1){\mathsf{\Sigma }}^{*}{\mathbf{B}}^{*}+{\rho }_2({\mathbf{Z}}_2-{\mathsf{\Theta }}_2)+{\rho }_3({\mathbf{Z}}_3-{\mathsf{\Theta }}_3){\mathsf{\Sigma }}^{*}{\mathbf{B}}^{*}+{\rho }_4({\mathbf{Z}}_4-{\mathsf{\Theta }}_4){\mathsf{\Sigma }}^{*}{\mathbf{B}}^{*},\hfill \end{array}\right.\hfill \end{array}$$

(25)

where $\mathbf{I}\in {\mathbb{R}}^{M\times M}$ is the identity matrix. The optimal solution is obtained by the inverse problem ${\mathsf{\Phi }}^{★}=\mathsf{\Xi }{\mathsf{\Psi }}^{-1}$.

The updates of ${\mathbf{Z}}_1$ and ${\mathbf{Z}}_2$ in steps 5 and 6 of Algorithm 2 can be accomplished by employing the Gaussian denoiser ${\mathcal{D}}_{{\mathcal{R}}_{r,\alpha /{\rho }_1}}$ and ${\mathcal{D}}_{{\mathcal{R}}_{m,(1-\alpha )/{\rho }_2}}$ as regularization terms ${\mathcal{R}}_r$ and ${\mathcal{R}}_m$, respectively. In our experiments, we utilized the TV algorithm discussed in Section 2.4 or BM3D [22] for both denoisers of ${\mathcal{D}}_{{\mathcal{R}}_{r,\alpha /{\rho }_1}}$ and ${\mathcal{D}}_{{\mathcal{R}}_{m,(1-\alpha )/{\rho }_2}}$. These Gaussian denoisers can restore smooth reconstructed frames and DMD modes while effectively removing noise.

The updates of ${\mathbf{Z}}_3$ and ${\mathbf{Z}}_4$ in steps 7 and 8 of Algorithm 2 require the computation of the proximity operators for the indicator functions of ${\iota }_{{\mathcal{B}}_{\mathbf{Y},ϵ}^F}(·)$ and ${\iota }_{{\mathbb{R}}^{N\times M}}(·)$, which are equivalent to the metric projections onto them. Similar to Equation (13), the metric projection onto ${\mathcal{B}}_{\mathbf{Y},ϵ}^F$ is given by

$${\mathcal{P}}_{{\mathcal{B}}_{\mathbf{Y},ϵ}^F}\left(\mathbf{X}\right)=\left\{\begin{array}{cc}\mathbf{X},\hfill & \mathrm{if}\mathbf{X}\in {\mathcal{B}}_{\mathbf{Y},ϵ}^F,\hfill \\ \mathbf{Y}+ϵ\frac{(\mathbf{X}-\mathbf{Y})}{{\parallel \mathbf{X}-\mathbf{Y}\parallel }_F},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(26)

Then, the metric projection onto ${\mathbb{R}}^{N\times M}$ is given by ${\mathcal{P}}_{{\mathbb{R}}^{N\times M}}\left(\mathbf{X}\right)=\mathrm{real}\left(\mathbf{X}\right)$, where $\mathrm{real}\left(\mathbf{X}\right)$ is the real part of $\mathbf{X}$.

4. Experiments

To demonstrate the effectiveness of the proposed method, we applied it to several noisy videos and compared it with naive noise removal methods in which the TV and BM3D denoisers were applied directly to the video frames. We refer to these methods as “naive TV” and “naive BM3D”. These results were obtained by setting $\alpha$ to 1, and these methods did not consider any regularization for the DMD mode. By comparing with such naive noise removal approaches, we can confirm the effectiveness of our proposed method that considers regularization for the DMD mode, namely the effectiveness of explicitly applying denoisers to the DMD mode. We refer to our method with the TV denoiser and the BM3D denoiser as “Ours with TV” and “Ours with BM3D”.

Figure 1 shows the original video scenes used for experiments. Scene 1 and Scene 2 videos were captured by the authors, Scene 3 and Scene 4 videos were selected from the SBMnet dataset [35], and Scene 5 video was selected from the DAVIS dataset [36]. We extracted $M=10,20,30$ [frames] from the first frame of each video. For the sake of simplicity, a color video was converted to grayscale. The details of each scene are briefly summarized as follows:

We independently added AWGN with three intensities, i.e., $15/255,25/255$, and $35/255$, to input video frames. The visually best results with the proposed method were obtained by setting $ϵ=0.95\sqrt{NM{\sigma }^2}$ and adjusting the value of $\alpha$ from 0 to 1 in steps of $0.1$. For the quality metrics, we used the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) [37]. If the structure similarity between the input and reference images is high, the SSIM value is closer to 1 (for details, see [37]).

Table 1. Average PSNR comparison.

Scene	$\mathit{\sigma }$	$\mathit{M}=10$ [Frame]					$\mathit{M}=20$ [Frame]					$\mathit{M}=30$ [Frame]
		Noisy	Naive TV	Naive BM3D	Ours with TV  $\left(\mathsf{\alpha }\right)$	Ours with BM3D  $\left(\mathsf{\alpha }\right)$	Noisy	Naive TV	Naive BM3D	Ours with TV  $\left(\mathsf{\alpha }\right)$	Ours with BM3D  $\left(\mathsf{\alpha }\right)$	Noisy	Naive TV	Naive BM3D	Ours with TV  $\left(\mathsf{\alpha }\right)$	Ours with BM3D  $\left(\mathsf{\alpha }\right)$
1	15/255	24.62	32.29	34.14	32.81 (0.4)	34.64 (0.1)	24.62	32.27	33.06	32.77 (0.5)	34.64 (0.1)	24.62	32.27	33.05	32.70 (0.6)	34.59 (0.1)
	25/255	20.23	30.21	31.81	30.31 (0.8)	32.24 (0.3)	20.23	30.28	31.65	30.25 (0.9)	32.19 (0.3)	20.23	29.95	31.65	30.25 (0.9)	32.18 (0.3)
	35/255	17.45	28.53	30.50	28.06 (0.9)	30.64 (0.5)	17.45	28.53	30.49	28.06 (0.9)	30.63 (0.5)	17.45	27.56	30.36	26.82 (0.9)	30.59 (0.6)
2	15/255	24.66	28.37	29.46	28.96 (0.5)	29.55 (0.4)	24.66	28.36	29.50	28.95 (0.6)	29.56 (0.6)	24.66	28.37	28.78	28.88 (0.8)	29.25 (0.1)
	25/255	20.32	26.34	27.47	26.55 (0.7)	27.48 (0.8)	20.32	26.52	27.00	26.54 (0.9)	27.28 (0.4)	20.32	26.29	27.04	26.02 (0.9)	27.32 (0.4)
	35/255	17.55	25.18	26.09	25.16 (0.9)	26.14 (0.5)	17.55	25.02	26.00	23.91 (0.9)	26.14 (0.5)	17.54	24.34	26.03	22.55 (0.9)	26.18 (0.5)
3	15/255	24.76	31.85	34.80	32.54 (0.4)	35.83 (0.1)	24.76	31.85	34.77	32.45 (0.6)	35.71 (0.2)	24.76	32.23	34.73	32.44 (0.6)	35.68 (0.1)
	25/255	20.44	29.25	32.74	29.63 (0.8)	33.08 (0.3)	20.44	29.57	32.74	29.61 (0.9)	33.07 (0.3)	20.45	29.41	32.70	29.09 (0.9)	33.03 (0.3)
	35/255	17.64	27.57	30.80	27.28 (0.9)	30.91 (0.7)	17.65	26.92	30.73	26.36 (0.9)	30.86 (0.6)	17.65	25.60	30.72	24.92 (0.9)	30.82 (0.6)
4	15/255	24.75	26.84	28.49	28.10 (0.3)	28.55 (0.4)	24.74	26.90	28.51	28.02 (0.5)	28.57 (0.6)	24.74	26.91	28.51	28.03 (0.6)	28.55 (0.7)
	25/255	20.43	24.31	25.66	25.17 (0.5)	25.67 (0.7)	20.43	24.93	25.27	25.16 (0.8)	25.27 (0.2)	20.43	24.93	25.28	25.16 (0.8)	25.40 (0.2)
	35/255	17.66	23.16	23.30	23.42 (0.8)	23.49 (0.3)	17.66	23.42	23.25	23.40 (0.9)	23.51 (0.3)	17.66	23.24	23.25	23.01 (0.9)	23.50 (0.3)
5	15/255	24.71	30.00	31.22	30.41 (0.6)	32.56 (0.2)	24.70	29.99	31.12	30.41 (0.7)	32.46 (0.2)	24.70	30.34	31.12	30.20 (0.9)	32.45 (0.2)
	25/255	20.41	27.43	29.26	27.68 (0.8)	29.67 (0.4)	20.39	27.67	29.18	27.57 (0.9)	29.66 (0.3)	20.38	27.09	29.17	26.70 (0.9)	29.65 (0.4)
	35/255	17.64	25.92	27.55	25.85 (0.9)	27.70 (0.7)	17.61	24.81	27.67	24.29 (0.9)	27.72 (0.9)	17.61	23.33	27.66	22.80 (0.9)	27.71 (0.6)

and

Table 2. Average SSIM comparison.

Scene	$\mathit{\sigma }$	$\mathit{M}=10$ [Frame]					$\mathit{M}=20$ [Frame]					$\mathit{M}=30$ [Frame]
		Noisy	Naive TV	Naive BM3D	Ours with TV  $\left(\mathsf{\alpha }\right)$	Ours with BM3D  $\left(\mathsf{\alpha }\right)$	Noisy	Naive TV	Naive BM3D	Ours with TV  $\left(\mathsf{\alpha }\right)$	Ours with BM3D  $\left(\mathsf{\alpha }\right)$	Noisy	Naive TV	Naive BM3D	Ours with TV  $\left(\mathsf{\alpha }\right)$	Ours with BM3D  $\left(\mathsf{\alpha }\right)$
1	15/255	0.4092	0.8536	0.8857	0.8618 (0.5)	0.8953 (0.1)	0.4094	0.8536	0.8620	0.8609 (0.6)	0.8955 (0.1)	0.4095	0.8534	0.8617	0.8590 (0.7)	0.8933 (0.1)
	25/255	0.2383	0.8042	0.8374	0.8044 (0.9)	0.8502 (0.3)	0.2381	0.8017	0.8323	0.7909 (0.9)	0.8484 (0.2)	0.2381	0.7613	0.8324	0.7904 (0.9)	0.8480 (0.2)
	35/255	0.1591	0.7215	0.8077	0.6779 (0.9)	0.8135 (0.5)	0.1591	0.7222	0.8075	0.6783 (0.9)	0.8133 (0.5)	0.1592	0.6367	0.7929	0.5783 (0.9)	0.8107 (0.7)
2	15/255	0.6173	0.7465	0.7792	0.7865 (0.5)	0.7822 (0.4)	0.6197	0.7501	0.7833	0.7912 (0.6)	0.7854 (0.6)	0.6202	0.7514	0.7370	0.7887 (0.8)	0.7659 (0.1)
	25/255	0.4191	0.6513	0.6842	0.6800 (0.7)	0.6852 (0.8)	0.4226	0.6760	0.6446	0.6824 (0.9)	0.6687 (0.2)	0.4233	0.6757	0.6487	0.6641 (0.9)	0.6725 (0.2)
	35/255	0.3022	0.6031	0.5991	0.6084 (0.9)	0.6045 (0.5)	0.3062	0.6097	0.5939	0.5577 (0.9)	0.6114 (0.5)	0.3066	0.5795	0.5982	0.4961 (0.9)	0.6159 (0.5)
3	15/255	0.4541	0.8882	0.9123	0.8918 (0.6)	0.9241 (0.1)	0.4541	0.8879	0.9118	0.8903 (0.8)	0.9216 (0.1)	0.4556	0.8868	0.9115	0.8900 (0.8)	0.9218 (0.1)
	25/255	0.2913	0.8449	0.8907	0.8445 (0.9)	0.8966 (0.3)	0.2913	0.8394	0.8897	0.8263 (0.9)	0.8950 (0.5)	0.2925	0.7969	0.8891	0.7633 (0.9)	0.8940 (0.3)
	35/255	0.2082	0.7530	0.8658	0.7108 (0.9)	0.8679 (0.5)	0.2081	0.6728	0.8639	0.6200 (0.9)	0.8661 (0.5)	0.2088	0.5611	0.8554	0.5139 (0.9)	0.8651 (0.5)
4	15/255	0.7696	0.8540	0.8963	0.8876 (0.4)	0.8965 (0.8)	0.7676	0.8545	0.8966	0.8821 (0.5)	0.8969 (0.6)	0.7678	0.8545	0.8966	0.8863 (0.6)	0.8972 (0.1)
	25/255	0.6069	0.7631	0.8174	0.8062 (0.5)	0.8178 (0.1)	0.6045	0.7955	0.7971	0.8050 (0.8)	0.8175 (0.1)	0.6043	0.7952	0.7971	0.8047 (0.8)	0.8076 (0.2)
	35/255	0.4835	0.7177	0.7058	0.7365 (0.8)	0.7530 (0.2)	0.4813	0.7350	0.7032	0.7347 (0.9)	0.7526 (0.2)	0.4810	0.7255	0.7031	0.7138 (0.9)	0.7523 (0.2)
5	15/255	0.5465	0.8786	0.8990	0.8810 (0.7)	0.9167 (0.1)	0.5442	0.8766	0.8962	0.8774 (0.9)	0.9141 (0.1)	0.5457	0.8511	0.8956	0.8350 (0.9)	0.9140 (0.1)
	25/255	0.3851	0.8186	0.8663	0.8142 (0.9)	0.8761 (0.4)	0.3821	0.7947	0.8628	0.7719 (0.9)	0.8734 (0.3)	0.3819	0.7175	0.8612	0.6849 (0.9)	0.8725 (0.3)
	35/255	0.2914	0.7450	0.8319	0.7178 (0.9)	0.8341 (0.7)	0.2883	0.6049	0.8228	0.5676 (0.9)	0.8310 (0.6)	0.2874	0.5102	0.8205	0.4828 (0.9)	0.8292 (0.5)

show the average PSNR and SSIM of all frames obtained by the proposed and naive methods. In the cases of “Ours with TV” and “Ours with BM3D”, the values of $\alpha$ that yielded the best results are shown in brackets. One can see from Table 1 that “Ours with BM3D” has the highest average PSNR values compared to naive TV, naive BM3D, and “Ours with TV”. Then, one observes from Table 2 that “Ours with BM3D” has higher average SSIM values than the other methods in most cases. However, in Scene 2, “Ours with TV” tends to have higher values than the other methods. This is because BM3D cannot preserve the complex texture of the background concrete wall, and it is lost due to over-smoothing. Thus, the SSIM values of BM3D denoiser-based methods tend to be lower than those of TV denoiser-based methods. It has been observed that “Ours with TV” has higher PSNR and SSIM values than naive TV, regardless of noise intensity in the cases of $M=10$ and 20. However, in the case of $M=30$ and $\sigma =35/255$, the values of “Ours with TV” are lower than those of naive TV. As frames increase, the DMD algorithm yields more DMD modes, including high-frequency modes representing fine vibrational components. The TV denoiser is suitable for improving spatial smoothness but does not preserve textures like repeating patterns. The estimated DMD modes by “Ours with TV” have fewer high-frequency components, so PSNR and SSIM values deteriorate.

Figure 2a,b illustrate the PSNR trends for “Ours with TV” under AWGN with $\sigma =15/255$ and $35/255$, respectively. They indicate that as the frame count rises, particularly under high noise levels, the performance of “Ours with TV” declines due to the increasing complexity of the DMD mode. The TV denoiser struggles to preserve minor DMD mode changes. Figure 2c,d demonstrate that “Ours with BM3D” maintains consistently high PSNR values, even with more frames and elevated noise levels. This stability is attributed to BM3D’s ability to exploit similar patches in the high-frequency DMD mode effectively. Similar trends were observed in the average SSIM values.

Figure 3 shows some close-ups of Scenes 1 and 4 degraded by AWGN with the standard deviation $\sigma =25/255$. In Scene 1, “Ours with BM3D” and naive BM3D can remove noise, preserving the human silhouette, while “Ours with BM3D” shows superior preservation of wood fine texture. However, “Our with TV” and naive TV only partially restore sharp edges and textures. In Scene 4, “Ours with BM3D” and naive BM3D successfully remove noise while maintaining the car and wood silhouette, whereas “Ours with TV” and naive TV cannot restore sharp edges. Notably, “Ours with BM3D” outperforms naive BM3D in restoring tree edges and textures. Its direct noise removal in the DMD mode effectively preserves high-frequency modes even in areas with motion and complex textures.

Figure 4 shows some close-ups of the DMD mode of Scene 1 in the case of AWGN with $\sigma =25/255$. This figure shows that “Ours with BM3D” can remove noise while preserving the edges and rich textures of the DMD modes ${\mathsf{\Phi }}_1$ and ${\mathsf{\Phi }}_{20}$. The restoration results of naive BM3D can effectively remove noise similar to “Ours with BM3D”. However, due to the implicit restoration of the DMD mode, its restoration accuracy seems inferior to “Ours with BM3D”. In contrast, “Ours with BM3D” can restore the textures of the DMD mode more clearly than naive BM3D. This is because noise removal can be applied directly to the DMD mode, resulting in preserved edges and textures. Although “Ours with TV” can reduce noise better than naive TV, it is less effective at removing noise than both methods based on the BM3D denoiser, especially in the high-frequency mode ${\mathsf{\Phi }}_{20}$.

Next, we applied “Ours with BM3D” and “Ours with TV” to real noisy video captured with a high ISO setting in low light conditions and compared the results with those of naive TV and naive BM3D to show their effectiveness on real video. Figure 5 shows some close-ups of the resulting images with the gamma correction set to $\gamma =1.3$ for better visibility. This figure shows that “Ours with BM3D” effectively preserves the edge of the fence and has a better noise removal effect than naive TV, naive BM3D, and “Ours with TV”. Conversely, “Ours with TV” is better at preserving complex details, such as concrete wall patterns that are difficult to recover with the BM3D denoiser.

Finally, we discuss the computational cost of the proposed method. All experiments were conducted using MATLAB R2021a on a system equipped with an AMD EPYC 7402P 2.80 GHz processor and 128 GB RAM. Our method uses the iterative algorithm based on the PnP-ADMM framework, where each iteration requires a denoiser computation. The TV denoiser requires iterative computation using the ADMM algorithm (see Section 2.4 for details). Fortunately, as shown in the reference [21], it is possible to perform fast computations using the fast Fourier transform to solve the quadratic minimization problem with respect to $\mathbf{x}$ in Equation (16). Therefore, this denoiser can be executed in a relatively short computation time. The BM3D denoiser requires iterative computation using a non-local mean algorithm. Specifically, it is necessary to repeatedly search for many patches similar to the target patch from a wide range around it, which requires high computational cost and relatively long execution time. Figure 6 shows the average computational time when naive TV, naive BM3D, “Ours with the TV”, and “Ours with BM3D” on a video with an image size of $256\times 256$. The execution time when applying each method to a video with a frame size of 10 was as follows:

• Naive TV was performed in less than 10 s.
• Naive BM3D was performed in about 25 s.
• “Ours with TV” was performed in about 20 s and shorter execution time than naive BM3D.
• “Ours with BM3D” was performed in about 75 s and about three times slower than naive BM3D.

“Ours with BM3D” achieves the highest noise removal accuracy among the compared methods. However, it also requires a longer computational time. Despite taking roughly three times longer than naive BM3D, we still consider it practical. Furthermore, the figure illustrates that the computation time for each method increases proportionally with the number of frames.

5. Conclusions

In this paper, we introduced a novel noise removal method for the DMD mode of a noisy video. Specifically, the minimization problem that simultaneously reduces the noise of the DMD mode and the reconstructed video was defined. Then, we solved the proposed problem using the PnP-ADMM algorithm. The experiments confirmed that the proposed method can effectively remove noise in the DMD mode and the reconstructed video. These results suggest the potential to provide more reliable results in image recognition and object detection, especially in video surveillance and object tracking applications where foreground and background separation is essential. The proposed method consistently demonstrated effectiveness over naive noise removal methods throughout the experiments.

In future work, we will employ stochastic gradient descent algorithms to improve the computational efficiency of the proposed PnP-ADMM algorithm. We will also apply the proposed method to other high-dimensional volume data noise removal problems, e.g., hyperspectral and CT/MRI imaging.