Single Image Deblurring for Pulsed Laser Range-Gated Imaging System with Multi-Slice Integration

Abstract

The multi-slice integration (MSI) method is one of the approachs to extend the depth of view (DOV) of the pulsed laser range-gated imaging (PLRGI) system. When the DOV is large enough and exceeds the depth of focus of the system, it may make some targets in the image clear and others blurred. In addition, forward scatter is also considered to have a blurring effect on the image. There is very little literature to solve the combined effect of forward scatter and defocus. An imaging model is built based on the model from Jaffe–McGlamery and Fourier optics. According to the imaging model, backscattered light is independent from reflected light from the target, and forward scatter has a relationship with the reflected light. Thus, backscattered light should be removed before deblurring. First, rolling ball and intensity transformation are used to remove the backscattered light and enhance the image. Then, a deep learning model based on Transformer is used to deblur the image. To enable the deep learning model to accommodate different degrees of blurred image, 16 different blur kernels are generated according to the imaging model. Sharp images from a DPDD dataset were chosen to train the model. Images of varying degrees of blur were collected from a water tank and a boat tank by the PLRGI system as test sets. Image deblurring results show that the proposed method can remove different levels of blur and can deal with images which have sharp targets and blurred targets together.

1. Introduction

Pulsed laser range-gated imaging (PLRGI) is one of the most effective methods to achieve underwater high-resolution imaging [1,2,3]. The conventional PLRGI system is mostly used for in situ observation for the limitation of the depth of view (DOV). To extend the DOV, many methods can be used. One is multi-channel receivers, which can acquire images from multi-slice at the same time [4,5]. However, this will make the system too complex and expensive. Another one is the multi-pulse integration (MPI) method, which splits the detection area into multi-areas and assigns pulses to detect each area alone and combines them to a single image [6,7]. Nevertheless, the number of pulses allocated to each slice is the same in the literature, and this makes it similar to a single slice with a large DOV. The MPI method introduces the problem that the distant target is much darker than the near target. Thus, we propose the multi-slice integration (MSI) method in the literature [8]. Different from the MPI method, the number of the pulses assigned to each slice and gate width (GW) of each slice can be different from each other in the MSI method. In this case, by adjusting the system parameters, the intensity of targets at different distances can be approximately equal. When the DOV is extended, the distance between multiple targets in the field of view may exceed the depth of focus (DOF), resulting in clear imaging of some targets and blurring of others (same as blur caused by improper focal length setting). In addition, forward scatter in water also has an effect on the blurring of the image [9]. Similar to other optical systems, the motion of the system or target can also cause image blur. Since motion blur is different from the above two blurring effects, it is not considered in this paper.

Forward scatter changes the direction of the light and blurs the image of the target. Various solutions have been proposed to address the blurring caused by the forward scatter. One is to build a model related to the shape and background of the target [10,11,12], and use the model to estimate the intensity and distribution of forward scattering light. The key point of these model-based methods is the accurate estimation of medium transmission [13,14,15]. However, these methods always are unstable and sensitive for three reasons: (1) image recovery is an ill-posed problem, (2) the underwater environment is complex and estimating underwater imaging parameters is difficult, and (3) underwater imaging models may be inaccurate in some cases. Another one is to deconvolve the blurred image using an iterative approach [9,16]. However, this approach relies on the initial estimate and has a high computational cost. There are also methods for underwater image enhancement which come with the ability to remove forward scatter [15,17]. However, they mainly focus on color cast, low contrast or backscattered light removal.

For defocus blur, researchers focus on detection and segmentation of the defocus blur area [18], defocus map estimation [19] and defocus blur removal. The traditional approach for defocus blur removal is to estimate the point spread function (PSF) and then use a deconvolution method to restore a clear image. However, the PSF is hard to estimate for the same reasons of model-based methods. In recent years, researchers have preferred to use deep learning methods to remove defocus blur. This approach uses an end-to-end network architecture that skips complex modeling problems and enables direct mapping of blurred images to clear images. Abuolaim [20] proposed DPDNet for removing defocus blur. However, the method is limited by the hardware, and the performance of the single image deblurring method needs to be improved. Son [21] proposed a kernel-sharing parallel atrous convolutional (KPAC) block for single image defocus deblurring. Quan [22] proposed a pixel-wise Gaussian kernel mixture deep neural network (GKMNet) for single image defocus blur removal. However, their performances on severely blurred regions is not that satisfactory. All the above networks are designed based on convolutional neural networks. Convolution is a local operation that usually only models the relationship between neighboring pixels. Transformer, on the other hand, is a global operation that models the relationship between all pixels. As a result, image restoration using Transformer is now at the forefront [23,24].

For underwater range-gated imaging, a blurred image is caused by the combined effect of forward scatter and defocus. Most of the literature focuses on removing the effects of forward scatter. There is very little literature to solve the combined effect of forward scatter and defocus blur due to the small DOV of the PLRGI system. In this paper, a two-step method is proposed to deblur the image collected by the PLRGI system with an MSI method. The main contributions of this paper are highlighted as follows.

• A two-step method is proposed to deblur the blurred image caused by the combined effect of forward scatter and defocus.
• An imaging model is built to generate training data for the network, and this can enable the model to deal with different levels of blurred image. It makes use of merits of both knowledge of underwater imaging and deep neural networks.
• Our method outperforms two state-of-the-art methods on several experiments under different water conditions in terms of both visual quality and quantitative metrics.

The paper is organized as follows: following this introduction section, principles of the MSI method are described, and an imaging model is built based on the model from Jaffe–McGlamery and Fourier optics. In Section 3, a two-step method is proposed to deblur the image according to the imaging model. First, rolling ball and intensity transformation are used to remove the backscattered light and enhance the image. Then, a deep learning model based on Transformer is used to deblur the image. In Section 4, in order to validate the method proposed, experiments were carried out to obtain the blurred image from the PLRGI system, and the experimental results are presented and discussed. Conclusions are drawn in the last section.

2. Theory

2.1. Multi-Slice Integration Method for PLRGI System

The PLRGI system enables a slice imaging approach similar to that of computed tomography. Position and width of the slice depends on the delay time and gate width. Intensity of the slice depends on the number of pulses assigned to the slice. To illustrate the principle of the MSI method, we take three slices, for example, which are shown in Figure 1. In the system, a laser with a high pulse repetition frequency is in use. The laser pulses in one video frame can be split into three groups, and each group corresponds to one slice. Each slice has its own delay time and gate width.

The timing sequence of the MSI method is shown in Figure 1b. The number of pulses assigned to target ①, ② and ③ are n₁, n₂ and n₃, respectively. Delay time of three groups are t₁, t₂ and t₃, which are different from each other. Every pulse has a range intensity profile (RIP), and pulses belonging to same group have the same RIP. Finally, three sub-RIPs are integrated in a frame and generate an image with an integrated RIP.

RIP of a single pulse can be expressed as,

$$E_i(z)=\left\{\begin{array}{ll}(\frac{2z}{c_w}-t_i+{\tau }_P)\Phi ,\hfill & \frac{{2z}^{\prime }}{c_w}\in [-{\tau }_p,0]\hfill \\ {\tau }_P\Phi ,\hfill & \frac{{2z}^{\prime }}{c_w}\in [0,{\tau }_i-{\tau }_p]\hfill \\ ({\tau }_i+t_i-\frac{2z}{c_w})\Phi ,\hfill & \frac{{2z}^{\prime }}{c_w}\in [{\tau }_i-{\tau }_p,{\tau }_i]\hfill \end{array}\right.$$

(1)

where τ_P is the pulse width, c_w is speed of the laser in water, t_i is the delay time of the gate, τ_i is gate width, z is the distance of the target, z′ = z-c_wt_i/2, and Φ is the laser pulse radiant flux that reach the camera. More details about the integrated RIP can be obtained from the literature [8].

Integrated RIP in a frame is,

$$E_{integrated}={\displaystyle \sum _{k=1}^3{n}_k{E}_k}$$

(2)

where k means kth slice, and E_k can be obtained from Equation (1).

2.2. Imaging Model Based on Jaffe-McGlamery

The commonly used underwater imaging model was developed by Jaffe [25] after improving on the model proposed by McGlamery [26]. According to the model, the total irradiance E_total that reaches the sensor consists of three components: light reflected from the target, forward scattering light from the reflected light and backscattered light from water. E_total can be expressed as,

$$E_{total}=E_{t\mathit{arg}et}+E_{fs}+E_{bs}$$

(3)

Here E_target equals E_integrated for the MSI method. E_fs and E_bs denote energy of forward scattering light and energy of backscattered light, respectively.

According to the Fourier optics, light energy output from CCD equals the convolution of E_total and the modulation transfer function (MTF) of the receiver MTF_sys and can be expressed as,

$$E_{out}=E_{total}\ast F^{-1}{(\mathrm{MTF}}_{\mathrm{sys}})$$

(4)

where F⁻¹ is the inverse Fourier Transform.

From Equations (3) and (4), we can acquire,

$$E_{out}=E_{t\mathit{arg}et}\ast F^{-1}{(\mathrm{MTF}}_{\mathrm{sys}})+E_{fs}\ast F^{-1}{(\mathrm{MTF}}_{\mathrm{sys}})+E_{bs}\ast F^{-1}{(\mathrm{MTF}}_{\mathrm{sys}})$$

(5)

For forward scattering light, E_fs can be calculated by the convolution operation of reflected light and the PSF of water. There are many models for the PSF of water. Here we take the form of the PSF of water from Hou [27],

$${\mathrm{PSF}}_{\mathrm{water}}=K\left({\theta }_0\right)\frac{br{e}^{-\tau }}{2\pi {\theta }^m}=K\left({\theta }_0\right)\frac{{\omega }_0\tau e^{-\tau }}{2\pi {\theta }^m}$$

(6)

where θ is the scattering angle, K(θ₀) is a constant, and m = 1/ω₀ − 2τθ₀. Note that τ is the optical length and defined as τ = cr. Here c is the total attenuation coefficient, and r is the distance from the system.

Thus, energy of the forward scattering light can be expressed as,

$$E_{fs}=E_{t\mathit{arg}et}\ast {\mathrm{PSF}}_{\mathrm{water}}$$

(7)

Substituting Equation (7) into Equation (5), we obtain,

$$E_{out}=E_{t\mathit{arg}et}\ast (F^{-1}{(\mathrm{MTF}}_{\mathrm{sys}})+{\mathrm{PSF}}_{\mathrm{water}}\ast F^{-1}{(\mathrm{MTF}}_{\mathrm{sys}}\left)\right)+E_{bs}\ast F^{-1}{(\mathrm{MTF}}_{\mathrm{sys}})$$

(8)

The composition of the receiver is shown in Figure 2. It consists mainly of an optical lens, a photocathode, a micro-channel plate (MCP), a fluorescent screen and a CCD.

According to the literature [28,29], MTF of the receiver can be acquired by,

$${\mathrm{MTF}}_{\mathrm{sys}}={\mathrm{MTF}}_{\mathrm{len}}\times {\mathrm{MTF}}_{\mathrm{MCP}}\times {\mathrm{MTF}}_{\mathrm{CCD}}$$

(9)

For a circular aperture without considering aberration, MTF_len can be expressed [30] as,

$$\left\{\begin{array}{l}{\mathrm{MTF}}_{\mathrm{len}}\left(f_x,f_y\right)=\frac{2}{\pi }(\phi -\cos \phi \cdot \sin \phi )\hfill \\ \phi ={\cos }^{-1}\left(\frac{\sqrt{f_x^2+f_y^2}}{f_c}\right)\hfill \\ f_c=\frac{D_0}{\lambda \times f_l}\hfill \end{array}\right.$$

(10)

where f_x, f_y denote the spatial frequencies in the x and y directions, respectively. D₀ is the diameter of the optical system, λ is the wavelength, and f_l is the focal length.

The MTF of the MCP [31] is expressed as,

$${\mathrm{MTF}}_{\mathrm{MCP}}=2\frac{\left|{\mathrm{J}}_1(2\pi f_{N}d)\right|}{\left|2\pi f_{N}d\right|}$$

(11)

where d is the size of the fine tubes in the MCP, and f_N is the spatial resolution.

The MTF of the CCD [30] can be expressed as,

$${\mathrm{MTF}}_{\mathrm{CCD}}\left(f_x,f_y\right)=\frac{\sin \left(\pi \alpha f_x\right)}{\pi \alpha f_x}\cdot \frac{\sin \left(\pi \beta f_y\right)}{\pi \beta f_y}$$

(12)

where

$$\left\{\begin{array}{l}\alpha ={\mu }_x/f_l\hfill \\ \beta ={\mu }_y/f_l\hfill \end{array}\right.$$

(13)

Here, μ_x and μ_y are the pixel size.

From Equations (8)–(13), we can calculate energy of all the light reaching the CCD. It can be seen from Equation (9) that MTF_sys will change as the focal length changes, and PSF_water will change as attenuation coefficient changes according to Equation (6). Thus, traditional model-based image restoration methods are very difficult to apply, for the model will change as the condition changes.

3. Method

We present the flowchart of our method in Figure 3. From Equation (8), we can see that energy of the light received by the CCD contains two parts: light from the target and backscattered light, and there is no correlation between these two terms. Thus, backscattered light can be removed before deblurring.

After backscattered light removing, Equation (8) can be changed to,

$$E_{out}=E_{t\mathit{arg}et}\ast (F^{-1}{(\mathrm{MTF}}_{\mathrm{sys}})+{\mathrm{PSF}}_{\mathrm{water}}\ast F^{-1}{(\mathrm{MTF}}_{\mathrm{sys}}\left)\right)$$

(14)

The right-hand side of the equation can be seen as a convolution of the reflected light with a blur kernel. Thus, Equation (14) can be used to generate training data for deep learning. Here, a Transformer model is used to deblur the image.

For backscattered light removal, several methods have been proposed. These approaches include lower-upper-threshold correlation [32], dark channel prior [33] and unsharp filtering [17]. The lower-upper-threshold correlation method does not remove the noise well. The dark channel prior method removes the noise while filtering out part of the target details. The unsharp filtering method maintains the target details, while the noise near the target is not effectively removed. In medical image processing, the rolling ball method is often used to remove background noise [34,35], which is both simple and effective. Thus, the rolling ball method is used to remove the backscattered light in this paper.

3.1. Backscattered Light Removal

The rolling ball method was first proposed by Sternberg [36]. This method draws the grayscale image as a 3D surface with the intensity as the third dimensional coordinate values. Then a 3D ball of a certain radius is used to roll over the 3D surface to form a series of tangent points, and these tangent points are interpolated and used as the background map. After that, background noise can be subtracted from the original image. Figure 4 shows an example of the rolling ball with line A–B.

Let the original image be represented by f(x,y), the rolling ball by b(x’,y’). Then background noise g(x_b,y_b) can be obtained from

$$g=(f\ominus b)\oplus b$$

(15)

Here

$$f\ominus b=\min \left\{f(x+x^{\prime },y+y^{\prime })-b(x^{\prime },y^{\prime })\left|(x^{\prime },y^{\prime })\in D_b\right.\right\}$$

and

$$f\oplus b=\max \left\{f(x-x^{\prime },y-y^{\prime })+b(x^{\prime },y^{\prime })\left|(x^{\prime },y^{\prime })\in D_b\right.\right\}$$

The corrected image t(x_t,y_t) can be obtained from

$$t=f-g$$

(16)

After removing the backscattered light, the brightness of the image is low. Thus, an intensity transformation should be used to adjust the brightness.

The histogram of the image is

$$p(r_i)=n_i,i=0,1,2\cdots ,L-1$$

(17)

where n_i is the number of pixels in the image whose gray value is r_i, and L is the gray level of the image.

Let

$$T(r_i)={\displaystyle \sum _{j=0}^{i}p(r_j)},i=0,1,2\cdots ,L-1$$

(18)

then the histogram cumulative distribution function is

$$D(i)=\frac{{\displaystyle \sum _{k=0}^{i}T(r_k)}}{{\displaystyle \sum _{k=0}^{L-1}T(r_k)}},i=0,1,2\cdots ,L-1$$

(19)

Considering the interference of bright spot and dark spot, the gray value in the range of 0.1–99.9% is selected as the valid range. Then the threshold value of transformation is

$$\left\{\begin{array}{cc}{D}_{\min }=D(k)& D(k-1)\le 0.1\%\le D(k)\\ D_{\max }=D(k)& D(k-1)\le 99.9\%\le D(k)\end{array}\right.$$

(20)

Let the transformed image be U(x,y), then the intensity transformation is

$$U(x,y)=\left\{\begin{array}{ll}0\hfill & f(x,y)\in [0,D_{\min })\hfill \\ \frac{t(x_t,y_t)-D_{\min }}{D_{\max }-D_{\min }}\hfill & f(x,y)\in [D_{\min },D_{\max }]\hfill \\ 255\times 0.9\hfill & f(x,y)\in (D_{\max },255]\hfill \end{array}\right.$$

(21)

3.2. Image Deblurring Using Transformer

There are many deep learning model architectures for defocus deblurring. A popular approach for image processing is the U-Net architecture [37], which is characterized by its encoder–decoder structure as well as skip connections.

In recent years, Transformer has been quite competitive compared to convolutional neural networks. After evaluating several model architectures, we have chosen the model by Wang [23], which employs U-Net architecture and Transformer block. The model architecture is shown in Figure 5. The model consists of three main modules: input (output) projection, Transformer block and down (up) sampling. The main purpose of the input projection is to convert the data into a format that can be processed by the Transformer block, and the output projection converts the processed data into images. Down sampling is used to compress the size of the feature map and reduce the computational cost, while up sampling is used to restore the compressed feature map to its original size. Skip connections in the model can preserve previously learned features across network layers.

The Transformer block is a key component of the network. It uses a window-based multi-headed self-attention (W-MSA) mechanism to map feature maps to different spaces and improve feature extraction. A locally enhanced feed-forward (LeFF) network is also used in the Transformer block to enhance the ability to capture local contextual information. The structure of the Transformer block is shown in Figure 6. Each Transformer block is composed of LayerNorm (LN) layer, W-MSA block, LeFF block and skip connections. The architecture of how W-MSA block works is shown in Figure 6b. It is able to improve the feature extraction by mapping to different spaces through QKV. All the cropped feature maps are compressed by a full connection (FC) layer and sent to the W-MSA block to calculate the attention value. The structure of the LeFF block is shown in Figure 6c. It is a new structure formed by adding a depthwise convolution to the Feed-Forward Network (FFN) of the standard Transformer block. FFN has limitations in capturing local contextual information and adding depthwise convolution to an FFN can enhance the ability to capture local contextual information.

The loss function used to train the model is

$$L(I^{\prime },I_{gt})=\sqrt{{‖I^{\prime }-I_{gt}‖}^2+{\epsilon }^2}$$

(22)

where I′ is the output image, I_gt is the ground-truth image, and ε (10⁻³) is a constant.

To train the model, sharp images are chosen from the DPDD dataset [20] and cropped to non-overlapping images of size 256 × 256 pixels. According to Equation (14), each cropped image is convolved with the system transfer function to generate the blur image. From Section 2.2, we know that MTF_sys will be changed when the system parameters change, and PSF_water will be changed when the condition of the water is changed. Thus, to enable the model to deal with different levels of blurred image, 16 different blur kernels are generated according to Equation (14) by changing the parameters of focal length and attenuation coefficient of the water.

4. Results and Discussion

4.1. Experimental Setup

To verify the methods in this paper, two experiments are carried out in a laboratory water tank and a boat tank at Huazhong University of Science and Technology. The imaging system used in the experiments and the experimental scenario are shown in Figure 7.

The PLRGI system used in the experiments is shown in Figure 7a. The laser works at a repetition rate of 10 kHz with a pulse length of 5 ns. The minimum gate width is 5 ns, and the maximum frame rate is 30 Hz. Some characteristics of the system are listed in

Table 1. Characteristics of the PLRGI system.

Specifications	Characteristics
Dimensions	Φ150 mm × L280 mm
Weight in water	≤1 kg
Depth range	Up to 200 m
Camera Lens	8–48 mm focal length
Field Of View	50°
Frame rate	1–30 Hz
Visual range	Up to 5 attenuation lengths

, and more information about the system can be acquired from the literature [8].

The water tank is 7 m long, 0.5 m high and 1 m wide. The size of the towing boat tank is 175 m × 6 m × 4 m (length × width × depth). The attenuation coefficient of the water was estimated [38] to 0.15 m⁻¹ and 0.25 m⁻¹, respectively.

4.2. Experiments in the Water Tank

Many images of varying degrees of blur were collected in a water tank as a test set, five of which are shown in Figure 8.

As can be seen in Figure 8, the contrast of the image is improved after the backscattered light is removed, but the blurring remains. To facilitate the description, the first row (the original data) of Figure 8 is denoted as ORI, the second row (data after the removal of backscattered light) as SUB, the third row (result of the deblurring of the first row) as ORI-deblur, and the fourth row (result of the deblurring of the second row) as SUB-deblur. Images of both ORI-deblur and SUB-deblur are sharper than those of ORI or SUB, and due to the influence of backscattered light, images of ORI-deblur are more blurred than those of SUB-deblur. Compared to ORI, the sharpness of the images of SUB-deblur increases significantly. In addition, we can see that the trained model can deblur images of different degrees of blur.

The Brenner gradient is often used to evaluate the sharpness of an image. It is expressed as,

$$f(I)=\sum _x\sum _y{\left[I\right(x+2,y)-I(x,y\left)\right]}^2$$

where I(x,y) is the gray value.

A comparison of images from Figure 8 by Brenner gradient is shown in Figure 9.

From Figure 9, we can see that the Brenner gradient of images from SUB-deblur are much larger than those from others, and this means that images from SUB-deblur are much sharper than those from others.

4.3. Experiments in the Boat Tank

Many images were collected in a boat tank by the PLRGI system with the MSI method, four of which are shown in Figure 10.

As can be seen in Figure 10, images of the first row were collected by the PLRGI system with large DOV, and some targets are out of focus. In addition, due to the large attenuation coefficient, backscattered light is obvious.

We can see from the second row that most of the backscattered light is removed, and the contrast of the images is improved. Images of both ORI-deblur and SUB-deblur are sharper than that of ORI or SUB, and due to the influence of backscattered light, images of ORI-deblur are not that clear. Compared to ORI, the sharpness of the images of SUB-deblur increases significantly. In addition, the trained model can deblur images of different levels of blur and can deal with images which have sharp targets and blurred targets.

A comparison of images from Figure 10 by Brenner gradient is shown in Figure 11.

From Figure 11, we can see that the Brenner gradient of images from SUB-deblur are much larger than those from others, and this means that images from SUB-deblur are much sharper than those from others. This is consistent with the subjective feelings.

4.4. Compare with Other Methods

We compare our method with the recent end-to-end deep-learning-based approach for defocus deblurring [21,22]. The result images are produced by using the pre-trained model provided by the authors. KPAC provides two versions of models, one is 2-level KPAC and the other is 3-level KPAC. We choose 2-level KPAC to deblur the images. For evaluation, we measure the Brenner gradient of the images.

Figure 12 shows a qualitative comparison. As the figure shows, our method produces sharper results with more details. KPAC is affected by noise severely and has a small effect on large-scale blur. GKMNet has little effect with large-scale blur.

Table 2. Brenner gradient of the images in Figure 12.

Image	Original-Img	KPAC (2-Level)	GKMNet	Proposed
Test 1	6,332,717	11,105,332	16,595,901	60,484,160
Test 2	1,859,769	2,309,726	4,788,044	12,987,654
Test 3	1,711,670	3,133,317	4,259,167	22,433,318

reports the quantitative comparison. As shown in the table, our method performs better than KPAC and GKMNet, which is consistent with the subjective feelings.

5. Conclusions

We have introduced an underwater image deblurring method for the PLRGI system with an MSI method. Our method considers the blur caused by the combined effect of forward scattering and defocus. According to the imaging model, we introduce the rolling ball and intensity transformation to remove backscattered light before deblurring, and we introduce a deep learning model based on Transformer to deblur images with different levels of blur. Extensive experiments on various water conditions were carried out, and the results show that the proposed method is effective and robust. The imaging model and the backscattered light removal method in this paper is also beneficial for other underwater vision tasks. Although the deep learning model based on Transformer has good performance in deblurring, it takes too much time and cannot handle this in real time. In order to be applied to the compact system, this issue will be studied in our future work.