Millimeter-Wave Image Deblurring via Cycle-Consistent Adversarial Network

Abstract

Millimeter-wave (MMW) imaging has a tangible prospect in concealed weapon detection for security checks. Typically, a one-dimensional (1D) linear antenna array with mechanical scanning along a perpendicular direction is employed for MMW imaging. To achieve high-resolution imaging, the target under test needs to keep steady enough during the mechanical scanning process since slight movement can induce large phase variation for MMW systems, which will result in a blurred image. However, in the scenario of imaging of a human body, sometimes it is difficult to meet this requirement, especially for the elderly. Such blurred MMW images would reduce the detection accuracy of the concealed weapons. In this paper, we propose a deblurring method based on cycle-consistent adversarial network (Cycle GAN). Specifically, the Cycle GAN can learn the mapping between the blurred MMW images and the focused ones. To minimize the effect of the shaking blur, we introduce an identity loss. Moreover, a mean squared error loss (MSE loss) is utilized to stabilize the training, so as to obtain more refined deblurred results. The experimental results demonstrate that the proposed method can efficiently suppress the blurring effect in the MMW image.

1. Introduction

Millimeter-wave (MMW) imaging has been widely employed in concealed weapons detection due to its penetration ability [1]. To obtain higher resolutions, the terahertz technology has also been studied for security inspections [2,3]. However, nowadays it is still not commercially applied because of the unstable performance of the circuits and the expensive manufacturing cost. In regards to the mostly applied MMW systems, the one-dimensional (1D) linear antenna array, with mechanical scanning along its perpendicular detection [4,5], is utilized for MMW imaging. Since mechanical scanning is involved, the target under test, such as a human body, is required to keep steady enough during the scanning process to eliminate unwanted phase variation. However, in practice, we could not avoid the slight movement, such as the shaking or swaying of the body, especially for the elderly, which will deteriorate the image quality. Since the state-of-the-art detection and classification methods usually apply the deep neural network architecture to realize the task through detecting the object edges or other characteristics in image, the blurred image with distorted object edges and features will definitely degenerate the detection accuracy [6]. The target detection from MMW images exhibits a similar characteristic. Thus, an efficient image deblurring approach is highly desired.

To achieve deblurring, Szeliski [7], Bar et al. [8] and Krishnan et al. [9] proposed some non-blind deblurring methods in optical applications, in which the Lucy–Richardson algorithm and Wiener filter were employed for deconvolution, assuming the blur kernels as a priori. However, the blur kernels cannot be accurately estimated in practical situations. Thus, the practicable deblurring algorithms should simultaneously estimate both the blur kernels and focused images [10]. More modified blind deconvolution methods [11,12,13,14] have been developed by using different point spread functions (PSFs) or prior information to reconstruct the focused images. Then, the blurring caused by the shaking of the optical lens can be eliminated. However, when the prior information is insufficient, it is difficult to determine the blur kernels. Thus, the classic non-blind and blind deblurring methods are hard to reconstruct satisfactory MMW images without blurring effects.

In addition to the aforementioned classic deblurring methods, the neural networks have also been employed in the field of optical blind deblurring. Sun et al. [15] employed the convolutional neural network (CNN) to estimate the blur kernel at the patch level, where each patch is assigned with a single motion label. However, this treatment violates the real data nature. Nah et al. [16] and Noroozi et al. [17] proposed an end-to-end deblur method without motion-blurred kernels by using multi-scale CNN; however, they can only handle mild Gaussian blur. Kupyn et al. [6] proposed a conditional adversarial network (cGAN) method to complete kernel-free blind motion deblurring. However, cGAN requires pairs of training samples, which are hard to obtain in practice. Thus, it is difficult to deal with real blurred samples. In addition, these methods all focused on the optical motion deblurring, which are not suitable for the MMW image deblurring.

Unlike the blurring induced by the shaking of lens, the MMW image blurring due to the movement of a human body, which is not a rigid object, may have different blur kernels with respect to different body parts. Clearly, this is more complex than the deblurring of optical images.

In this paper, we introduce the cycle-consistent adversarial network (Cycle GAN) [18] based method to realize the end-to-end deblurring of MMW imaging. The network can learn the translation between blurred MMW images and focused MMW images without using paired training samples. Different human shaking behaviors and the shake of different parts of the human body may produce different features of blur. To solve this problem, we collected MMW images of the human body under various shaking conditions for network training to improve network generalization. In addition, we adopted a mean squared error loss (MSE loss) in lieu of the binary cross entropy loss (BCE loss) to stabilize the model training procedure [19]. We also introduced an identity mapping loss function [20] to prevent the loss of information. The identity loss encourages mapping to keep the main information of the deblurred images the same as that of the blurred ones, which would fit well with the actual blurred MMW images. Additionally, we introduce the dropout operation during the generator training process to reduce the risk of overfitting. The final deblurred MMW images reconstructed by the proposed method show fair texture details without almost any artifacts.

The innovations of this paper are summarized as follows:

(a)

To the best of our knowledge, we are the first to address the phenomenon of MMW image blurring caused by the shaking of the human body, and propose an end-to-end deblurring architecture based on Cycle GAN.

(b)

The MSE loss and the identity loss are incorporated in Cycle GAN to ensure a more suitable model for the MMW deblurring task.

The rest of this paper is organized as follows. In Section 2, the model of the scattered waves is presented, as well as the imaging algorithm, to show the cause of the MMW image blurring. Then, the deblurring Cycle GAN architecture and the loss function are described in detail. In Section 3, the MMW imaging prototype is introduced. The deblurred results of the proposed method are shown in comparison with the classical blind deconvolution methods. In Section 4, the conclusions are drawn.

2. Formulation

2.1. Millimeter-Wave Imaging

The near field imaging scheme using a linear monostatic array with mechanical scanning is shown in Figure 1, where “●TR” denotes the position of the transceiver element. The scattering coefficient of each scatterer is represented by $\sigma (x,y,z)$. The demodulated scattered waves from the scatterers can be expressed as:

$$s(x’,y’,k)={\displaystyle \mathrm{\iiint }\sigma (x,y,z)e^{-j2kR}\mathrm{d}x}\mathrm{d}y\mathrm{d}z$$

(1)

where $R=\sqrt{{\left(x-x’\right)}^2+{\left(y-y’\right)}^2+{\left(z+R_0\right)}^2}$ is the round-trip distance between the target and the antenna, $k$ is the wavenumber with $k=\frac{2\pi f}{c}$, $f$ is the operating frequency, $c$ is the speed of light, and $R_0$ denotes the distance between the array aperture and the imaging plane.

The reconstruction of the target using RMA [2] can be expressed as:

$$\sigma (x,y,z)={\mathrm{FT}}_{3\mathrm{D}}^{-1}\left\{{\mathrm{IN}}_{1\mathrm{D}}\left\{{\mathrm{FT}}_{2\mathrm{D}}\left[s(x’,y’,k)\right]e^{\left(\mathrm{j}{k}_z{R}_0\right)}\right\}\right\}$$

(2)

where ${\mathrm{FT}}_{3\mathrm{D}}^{-1}\left\{\cdot \right\}$ represents the three-dimensional (3D) inverse Fourier transform, ${\mathrm{FT}}_{2\mathrm{D}}\left\{\cdot \right\}$ indicates the two-dimensional (2D) Fourier transform, ${\mathrm{IN}}_{1\mathrm{D}}$ denotes the 1D interpolation in the wavenumber domain in the loop of the 2D spatial frequency domain. According to (1), if the human body slightly shakes, the antennas will collect the signals with the information of different postures. This will lead to blurring of the MMW images. As a result, the blurred object in the MMW image would be difficult to recognize through the subsequent detection module.

2.2. Cycle GAN Methodology for MMW Image Deblurring

To remove the MMW image blurring, we utilized the end-to-end Cycle GAN method to transfer the blurred MMW radar images to the focused ones, without considering the difficult unknown priors and blur kernels.

The goal of the MMW deblurring Cycle GAN is to learn the mapping functions between the blurred MMW image dataset $A$ and the focused MMW image dataset $B$, given training samples ${\left\{a\right\}}_{i=1}^N$ and ${\left\{b\right\}}_{j=1}^M$, where $a_i\in A$ and $b_j\in B$. We denote the MMW data distribution as $a\sim \mathbb{P}\left(a\right)$ and $b\sim \mathbb{P}\left(b\right)$, where $\mathbb{P}\left(a\right)$ and $\mathbb{P}\left(b\right)$ represent the probability distributions of $a$ and $b$, respectively. As Figure 2 shows, the MMW deblurring Cycle GAN contains two mappings: generators $G:A\to B$ and $F:B\to A$. In addition, two adversarial discriminators, $D_A$ and $D_B$, with the same structure are introduced. $D_A$ aims to distinguish between the real blurred MMW image $\left\{a\right\}$ and the synthetic image $\left\{F\left(b\right)\right\}$. Similarly, $D_B$ aims to distinguish between the real focused MMW image $\left\{b\right\}$ and the deblurred MMW image $\left\{G\left(a\right)\right\}$. The structure of the generators $G$ and $F$ is shown in Figure 3. We mark the network with four modules. The part I is related to the forward convolution module. The second part II is related to stacked Resnetblock module. The third one and the fourth one are respectively referred to as the transpose convolution module and the convolution and reflection module. Different from the classical Cycle GAN, during the process of training the MMW deblurring Cycle GAN, we add a dropout [21] with the probability of 0.5 to each layer in ResnetBlocks to randomly deactivate the neurons with the probability of 0.5. The module can improve the generalization ability of the model and prevent overfitting. The neural networks that are fully activated by sigmoid have almost no sparsity and could miss some key information in the image, while ReLU has good sparsity due to its unilateral inhibition, and can also alleviate overfitting. Therefore, we chose ReLU as the activation function in this work. The discriminators $D_A$ and $D_B$ use the structure of classical PatchGANs [18].

The loss function of the classic Cycle GAN includes two terms: (1) the adversarial loss of GAN [22], which is used to update the parameters of the generator and discriminator; (2) the cycle consistency loss [16], which can prevent semantic loss and mode collapse. In the proposed MMW deblurring Cycle GAN, we further employ the identity loss [20] to maintain the detail information in the deblurred images, which is an incremental module compared with the classic Cycle GAN.

In regard to the adversarial loss, we employ MSE loss instead of BCE loss, because the former is more stable during the training procedure and generates better deblurred MMW images. The adversarial loss is denoted as follows:

$$\begin{array}{cc}{\mathcal{L}}_{adv}\left(G,D_B,A,B\right)\hfill & ={\mathbb{E}}_{a\sim \mathbb{P}\left(a\right)}\left[{\left(D_B\left(G\left(a\right)\right)-1\right)}^2\right]\hfill \\ & +{\mathbb{E}}_{b\sim \mathbb{P}\left(b\right)}\left[{\left(D_B\left(b\right)-1\right)}^2\right]+{\mathbb{E}}_{a\sim \mathbb{P}\left(a\right)}\left[D_B{\left(G\left(a\right)\right)}^2\right]\hfill \end{array}$$

(3)

where $G$ aims to minimize the adversarial loss, while $D_B$ aims to maximize it. $\mathbb{E}\left[\cdot \right]$ means the mathematical expectation of the inner function. The adversarial loss for the mapping function $F$ can be expressed in the same way as ${\mathcal{L}}_{adv}\left(F,D_A,B,A\right)$.

$$\begin{array}{cc}{\mathcal{L}}_{adv}\left(F,D_A,B,A\right)\hfill & ={\mathbb{E}}_{b\sim \mathbb{P}\left(b\right)}\left[{\left(D_A\left(F\left(b\right)\right)-1\right)}^2\right]\hfill \\ & +{\mathbb{E}}_{a\sim \mathbb{P}\left(a\right)}\left[{\left(D_A\left(a\right)-1\right)}^2\right]+{\mathbb{E}}_{b\sim \mathbb{P}\left(b\right)}\left[D_A{\left(F\left(b\right)\right)}^2\right]\hfill \end{array}$$

(4)

The second term is a cycle consistency loss, which is used to reduce the space of possible mapping functions [18] and prevent the information of the original blurred MMW image. As shown in Figure 2, each blurred MMW image $a$ from the domain $A$ should be brought to the original blurred one by the forward deblurring translation cycle, i.e., $a\to G\left(a\right)\to a_{rec}\approx a$, where $a_{rec}=F\left(G\left(a\right)\right)$. Similarly, the same representation is given for focused MMW images from the domain $B$: $b\to F\left(b\right)\to b_{rec}\approx b$, where $b_{rec}=G\left(F\left(b\right)\right)$. In the training process, we introduce the weights ${\lambda }_a$ and ${\lambda }_b$ to the forward and backward cycle consistency loss, respectively. They can control the emphasis of blurred MMW images to focused MMW images. This behavior can be represented as the cycle consistency loss:

$$\begin{array}{cc}{\mathcal{L}}_{cyc}\left(G,F\right)\hfill & ={\lambda }_a{\mathbb{E}}_{a\sim \mathbb{P}\left(a\right)}\left[{‖\left(F\left(G\left(a\right)\right)-a\right)‖}_1\right]\hfill \\ & +{\lambda }_b{\mathbb{E}}_{b\sim \mathbb{P}\left(b\right)}\left[{‖\left(G\left(F\left(b\right)\right)-b\right)‖}_1\right]\hfill \end{array}$$

(5)

To enhance the feature extraction ability of the network and reserve the detail information during the transformation, we attach the identity loss [17] to the mapping function $G$. The identity loss can be expressed as:

$${\mathcal{L}}_{identity}\left(G\right)={\mathbb{E}}_{b\sim \mathbb{P}\left(b\right)}\left[{‖\left(G\left(b\right)-b\right)‖}_1\right]$$

(6)

Now, the total loss function of the MMW deblurring Cycle GAN can be expressed as:

$$\begin{array}{cc}\hfill \mathcal{L}& ={\mathcal{L}}_{adv}\left(G,D_B,A,B\right)+{\mathcal{L}}_{adv}\left(F,D_A,B,A\right)\hfill \\ & +{\mathcal{L}}_{cyc}\left(G,F\right)+{\lambda }_{idt}{\mathcal{L}}_{identity}\left(G\right)\hfill \end{array}$$

(7)

Finally, the MMW image deblurring can be achieved.

3. Results

In this section, we utilized a MMW imaging prototype for data collection, and verified the efficiency of the proposed MMW image deblurring method. The working frequency was from 32 to 37 GHz. The prototype consists of a linear monostatic array with a horizontal mechanical scanning structure, as shown in Figure 4a. The transmit antenna worked in sequence, while the two neighboring receiving antennas collected the EM wave at the same time. This working pattern led to 383 virtual monostatic sampling points with an interval of 5 mm. The linear array scans horizontally to form a 2D aperture for 3D image reconstruction. The scanning spacing is 4 mm with 251 moving steps. The subject under test carried a concealed wrench standing in front of the array at a distance of 0.3 m, as shown in Figure 4. During the data collection, the subject stood firmly or shook slightly.

In the experiments, we generated 577 blurred MMW images (dataset $A$) and 612 focused MMW images (dataset $B$) as the training dataset. The dataset was collected under the scene that the subject is shaking slightly with varying degrees. The test data are collected under the scene where the subject is shaking slightly. The test dataset contains 80 blurred MMW images. The concealed objects (knife and wrench) hidden on the waist of the human body in the experiment are all covered by clothing. The size of all the employed images is $512\times 256\times 1$(height × width × channel).

The original reconstructed MMW images are shown in Figure 5a,f,k, representing the subject (taking a wrench) shaking vertically to the array, the subject (taking a knife) shaking parallel to the array and the subject (no object concealed) shaking parallel to the array, respectively. Although the human body shook slightly, the concealed objects and the human body exhibit seriously blurred properties, such as the invisible texture and coarse edge. We employ the classic blind deconvolution method [10], the original Cycle GAN [18] and the DeblurGAN [6], for comparison. In regard to the blind deconvolution method, the blur kernel was first estimated through the parts of the image, then was employed to restore the whole image, as shown in Figure 5b,g,l. Clearly, the image quality still significantly deteriorates with serious ring effects. Although this method may be suitable for the optical instruments, it fails in the situation of the target shaking in the scenario of MMW imaging. The deblurred images, by the original Cycle GAN, are shown in Figure 5c,g,l. Note that some detailed information is lost. In particular, the arms of the human body become much longer than the usual case. More seriously, the original Cycle GAN distorts the semantic information of the human body, and generates similar deblurred images for different blurred ones, which has the risk of mode collapse. The deblurred images by the DeblurGAN are demonstrated in Figure 5d,i,n. The results still show heavy blurred effect. In addition, there are a lot of disturbing clutters in the background.

Finally, the deblurred images from the proposed method are demonstrated in Figure 5e,j,o, which exhibit clearer finer textures than those obtained by the other state-of-the-art deblurring methods, such as the classic blind deconvolution method and the DeblurGAN method. As compared with the results of the original Cycle GAN, it proves that through incorporating the MSE loss and the identity loss, the proposed network obtains a more stable performance than the original Cycle GAN. Moreover, the proposed method achieves a more focused image with a clearer human outline and object texture in comparison to the results of the blind deconvolution in Figure 5b,g,l and the DeblurGAN in Figure 5d,i,n.

Furthermore, the image entropy is calculated to provide quantitative comparisons among the different methods. As is listed in

Table 1. Comparison of entropy of the results using different deblurring methods.

Objects	Blurred Image	Blind Deconvolution	Original Cycle	DeblurGAN	Proposed
Wrench(a–e)	0.7633	0.9805	0.6090	0.9943	0.6704
None(f–j)	0.9675	0.9824	0.6150	0.9972	0.6374
Knife(k–o)	0.6914	0.9627	0.6145	0.9801	0.6623

, the entropy of the original Cycle GAN possesses the lowest value; however, the deblurred results are distorted, as shown in Figure 5c,h,m. Excluding the original Cycle GAN, the entropy of the proposed method is the lowest in comparison to that of the classic blind deconvolution method and the DeblurGAN method, which further indicates the superiority of the proposed method. Comparing with the original distorted image, the entropy of the deblurred image by the proposed method decreases by 16.83% on average.

Then, we evaluate the computational complexity of the proposed method measured by the floating-point operations (FLOP) [23]. A single operation is referred to as one FLOP, such as one addition or one multiplication. To do so, we divide the generative architecture into four modules, as shown in Figure 3. Here we present the computational complexity of the first three remarkable network modules: convolutional layer, transpose convolutional layer and InstanceNorm. The computational complexity of the convolutional layer is calculated as:

$$C_{conv}=H_{out}\ast W_{out}\ast K^2\ast c_{in}\ast c_{out}$$

(8)

where $H_{out}$ and $W_{out}$ denote the height and width of the output image, respectively, $K$ denotes the size of the convolution kernel and $c_{in}$ and $c_{out}$ denote the numbers of input and output channels, respectively. For example, given a convolution layer with a $7\times 7$ convolution kernel where the sizes of the input and output data are $512\times 256\times 64$ and $256\times 128\times 128$, respectively, the computational cost is 13,153,337,344 FLOPs from (8).

The computational complexity of the transpose convolutional layer is calculated as:

$$C_{convT}=H_{in}\ast W_{in}\ast K^2\ast c_{in}\ast c_{out}$$

(9)

where $H_{in}$ and $W_{in}$ denote the height and width of the input image, respectively. For example, given a convolution layer with a $3\times 3$ convolution kernel where the sizes of the input and output data are $256\times 128\times 128$ and $512\times 256\times 64$, respectively, the computational cost is 2,415,919,104.

Finally, the InstanceNorm can be expressed as:

$$y=\frac{x-E\left[x\right]}{\sqrt{Var[x]+\epsilon }}\ast \gamma +\mathrm{\beta }$$

(10)

where $x$ is the input data, $E\left[x\right]$ is the expectation of $x$, $Var[x]$ is the variance of $x$, $\epsilon$ is a non-zero constant and $\gamma$ and $\beta$ are learnable parameter vectors of input size. Then, the computational complexity of the InstanceNorm is calculated by:

$$C_{IN}=8\ast H_{in}\ast W_{in}\ast c_{in}$$

(11)

For example, given the input data of $512\times 256\times 64$, respectively, the computational cost is 67,108,864 from (11).

The ReflectionPad module is concerned about the transformation of data size, therefore, no FLOP is required. In summary, the computational cost of each module (marked in Figure 3) of the proposed method is listed in

Table 2. The computational cost of different modules of the proposed deblurring architecture.

Module	Computations (FLOPs)
Ⅰ	27,099,402,788
Ⅱ	87,257,097,216
Ⅲ	5,140,119,552
Ⅳ	423,620,060

. The overall computational cost is approximately 0.12 TFLOPs. The computational cost of different methods used in this paper is calculated and listed in

Table 3. Comparison of computational cost of different deblurring methods.

Methods	Computations (FLOPs)
Blind deconvolution	0.006 T
Original Cycle	0.13 T
DeblurGAN	0.13 T
proposed	0.12 T

. The blind deconvolution is achieved using an Intel i7-11800H (maximum of 1.76 TFLOP per second) and does not use GPU acceleration, as it does not require GPU training. The proposed deblurring method, the original Cycle GAN and the DeblurGAN are achieved using Titan Xp (maximum of 12 TFLOP per second). Hence, the real-time image deblurring can be promised. The processing time of different methods is listed in

Table 4. Comparison of processing time of different deblurring methods.

	Blind Deconvolution	Original Cycle	DeblurGAN	Proposed
Time	2.367 s	0.338 s	0.322 s	0.299 s

. Clearly, the proposed method is the fastest one among the three methods.

4. Conclusions

In this paper, we propose an effective deblurring method for MMW images. The linear array associated with mechanical scanning may generate blurred reconstructed images due to the slight shaking of the human body during the data collection procedure. We utilized an architecture based on the Cycle GAN deal with the transformation from unpaired blurred MMW images to focused ones. The identity loss and the MSE loss are integrated to ensure the stable performance of the network. The experimental results demonstrated the superiority of the proposed deblurring method over both the classical blind deconvolution method and the original Cycle GAN without the introduced loss. In future work, we will prune the architecture of the proposed method to address a lightweight one as a plugin module in the post-processing of the MMW images.