A DeturNet-Based Method for Recovering Images Degraded by Atmospheric Turbulence

Abstract

Atmospheric turbulence is one of the main issues causing image blurring, dithering, and other degradation problems when detecting targets over long distances. Due to the randomness of turbulence, degraded images are hard to restore directly using traditional methods. With the rapid development of deep learning, blurred images can be restored correctly and directly by establishing a nonlinear mapping relationship between the degraded and initial objects based on neural networks. These data-driven end-to-end neural networks offer advantages in turbulence image reconstruction due to their real-time properties and simplified optical systems. In this paper, inspired by the connection between the turbulence phase diagram characteristics and the attentional mechanisms for neural networks, we propose a new deep neural network called DeturNet to enhance the network’s performance and improve the quality of image reconstruction results. DeturNet employs global information aggregation operations and amplifies notable cross-dimensional reception regions, thereby contributing to the recovery of turbulence-degraded images.

1. Introduction

Atmospheric turbulence is one of the main issues that cause image degradation when detecting objects at long ranges. Created by the random fluctuation of the refractive index, atmospheric turbulence is a spatial-temporal blur that cannot be measured directly [1,2,3]. With the development of adaptive optics, Noll et al. [4] established a relationship between turbulence and wavefront distortion, as described by the Zernike polynomials. Thus, a turbulence-degraded image can be efficiently restored using deconvolution algorithms when wavefront distortion or its point spread function (PSF) is correctly obtained [5,6,7,8,9].

Depending on whether the PSF is obtained in advance or not, the deconvolution algorithms can be divided into blind and non-blind deconvolution algorithms. Non-blind deconvolution algorithms restore degraded images using a known PSF. Typical non-blind deconvolution algorithms include inverse filtering, Wiener filtering, and other algorithms [10,11,12,13]. However, non-blind deconvolution algorithms need additional devices, such as a wavefront sensor (WFS), to detect the exact PSF, which makes the system complex [10,14]. Compared to non-blind deconvolution, blind deconvolution algorithms reconstruct degraded images in the presence of a poorly determined PSF, so it does not need any wavefront detection devices. However, because both the PSF and the initial image must be restored from the degraded images, blind deconvolution is an ill-posed problem. Thus, traditional blind deconvolution algorithms are mostly based on image sequences or iterative methods, like lucky imaging [15], iterative blind deconvolution [16], and their improvement algorithms [17,18].

In recent years, deep learning methods have received much attention in turbulence-blind deconvolution [19,20]. Compared with other methods, deep learning has considerable advantages in solving ill-posed and nonlinear problems for its end-to-end learning and data-driven approaches [21,22,23]. After training with a large amount of data, a deep neural network establishes a hidden nonlinear mapping between the input and output directly. Thus, deep learning methods can be properly used in ill-posed optical information restoration, such as holographic reconstruction [24,25], super-resolution imaging [26,27], image denoising [28,29], and phase extraction [30,31]. In addition, deep learning has also achieved good results in areas such as the modulation classification of signals [32,33,34,35,36], which proves the successful application of deep learning algorithms in various fields. Some researchers have also shown the effectiveness of deep learning in turbulence image reconstruction [37,38,39]. However, deep learning-based turbulent-degraded algorithms face some difficulties, such as the difficulty to obtain both the turbulence image and its PSF, the effect of noise, the complexity of the blur kernel, and the similarity of the recovered target information. Thus, we need to establish a new deep natural network, perform different kinds of experiments to obtain different data, and test the effectiveness of the network in both simulations and experiments.

In this paper, we propose a deep natural network called DeturNet for improving the images degraded by atmospheric turbulence. The advantage of this network is that it preserves global information and makes the image features better transferred by enhancing the interaction of information in different dimensions. In addition, this network shows good robustness based on the results and has some generalization ability with a small and single-scene dataset. In order to verify the performance of our network, we conducted simulations, as well as laboratory and outdoor experiments. Both simulation and experiment show that this method has a good ability for improving the degraded image in terms of turbulence removal and noise immunity and is also fast enough to be used in real-time applications.

2. Materials and Methods

2.1. Atmospheric Turbulence Imaging Model

In this section, the properties and a simplified model of atmospheric turbulence in long-range imaging are discussed in detail. Figure 1 shows a model of light propagating through atmospheric turbulence. In a high-F-number optical system, the PSF of the turbulence can be considered a spatially invariant function [40]. Thus, the observed degraded image can be modeled using Equation (1) [5].

$$g\left(r,\theta \right)=f(r,\theta )\otimes h(r,\theta )+n(r,\theta )$$

(1)

where $g(r,\theta )$ is the observed image, $f(r,\theta )$ represents the actual object, $h(r,\theta )$ represents the PSF, $n(r,\theta )$ represents the noise, $(r,\theta )$ represents space coordinates of planar image, and $\otimes$ denotes the convolution operation.

Then, the relationship between PSF and its wavefront distortion is shown in Equation (2)

$$h(r,\theta )={\left|ℱ\left\{P(r,\theta )\exp \left[i\phi (r,\theta )\right]\right\}\right|}^2$$

(2)

where P(r,θ) represents the optical pupil function of the telescope, $\phi (r,\theta )$ is the wavefront distortion, and $ℱ\left\{\right\}$ represents the Fourier transform. Since the optical pupil function is constrained in this paper, the PSF only depends on the wavefront distortion.

The phase $\phi (r,\theta )$ of the wavefront distortion caused by atmospheric turbulence can be decomposed in a Zernike polynomial orthogonal in the circular domain [4], as shown in Equation (3).

$$\phi (r,\theta )={\displaystyle \sum _{i=1}^{\infty }{a}_i{z}_i(r,\theta )}$$

(3)

where $a_i$ is the Zernike coefficient that represents the Zernike polynomial on the $i$th term. The relationship between the Zernike coefficient and the turbulence can be determined using [4]. According to the Kolmogorov turbulence theory, the relationship between the covariance matrix C = [c_ij] and the Zernike polynomial coefficient vector $A=\left\{a_1,a_2,\dots ,a_n\right\}$ can be obtained using Equation (4).

$$\langle a_i{a}_j\rangle =c_{ij}{\left(\frac{D}{r_0}\right)}^{5/3}$$

(4)

where c_ij is the covariance coefficient, D is the aperture of the telescope, and r₀ is the atmospheric coherence length. Thus, the coefficient matrix A can be derived from the Karhumen-Loeve polynomial, as shown in Equation (5).

$$\{\begin{array}{l}C=VS{V}^T\\ A=VB\end{array}$$

(5)

where V is the diagonal matrix, S is the Karhumen-Loeve polynomial coefficient matrix, and B represents the phase wavefront that can be considered as a Gaussian-distributed random vector with zero mean and variance array S. Based on the above analysis, we can obtain a Gaussian-type random vector distribution A with zero mean and make sure that the calculated wavefront distortion conforms to the Kolmogorov turbulence theory model.

2.2. DeturNet

To solve the ill-posed turbulence question, a deep learning method called DeturNet is proposed in this section, which is shown in Figure 2. Inspired by existing networks [41,42], DeturNet consists of two subnetworks, which are both U-Net structures. The two subnetworks connected in series have the following advantages: Since the output of the previous subnetwork as the input of the next subnetwork has an effect on the depth of the network, we believe that the deeper the depth of the network, the weaker the effect of atmospheric turbulence on the reconstruction, and the easier the network learns the feature distribution of the image, which is conducive to obtaining a better reconstruction; in addition to the effect of atmospheric turbulence, the design of the two subnetworks may be applicable to the for low-level and similar end-to-end tasks.

Each subnetwork has a five-layer structure that contains three parts: downsampling, upsampling, and skipping connections. In the downsampling stage, a HIN block is used to implement downsampling of the network and perform instance normalization operations, as shown in Figure 3a. In the upsampling stage, the Res Block is used to improve the reconstruction speed and to avoid gradient disappearance, as is shown in Figure 3b. Between down-sampling and up-sampling, a hopping connection is established by adding a cross-stage feature (CSFF) and a supervised attention module (SAM) [43]. The CSFF module allows features from one stage to be transformed and fused with features from the next stage, helping to enrich the multiscale features in the next stage, and the SAM helps to propagate useful features from the feature map to the next stage. Although these modules are good for information extraction and transmission, the above mechanism suffers from the loss of the channel and spatial local information of the image.

In order to preserve the channel and spatial local information and enhance the interaction between the different dimensions of information, we added a global attention mechanism (GAM) [44] to improve the utilization of multichannel information and amplify the global interaction representation, as shown in Figure 3c. The GAM comprises two submodules: a channel attention mechanism (as shown in Figure 4) and a spatial attention mechanism (as shown in Figure 5). The two-layer multilayer perceptron (MLP) is an encoder-decoder structure with a reduction rate r in the channel, which is used to amplify the spatial dependence of channels at different latitudes.

Through the design of a bipartite network, the depth of the network can be increased, the nonlinear fitting ability of the network can be improved, and the effect of the complex input transformations on the performance of the network can be learned more easily, which is conducive to better fitting the characteristics of the network. Aiming at the characteristics of atmospheric turbulence, the use of a baryon network can improve the ability of the model to recover the turbulence degradation image to a certain extent.

2.3. The Implement of DeturNet

In this section, the establishment of the training dataset and platform is proposed. As shown in Figure 6, we first selected 700 remote sensing images of aircraft from the NWPU-RESISC45 dataset [45] as the original image dataset and simulated 700 phase screens using the 4–60th Zernike polynomial from Equations (3)–(5) as the phase screen of atmospheric turbulence. The NWPU-RESISC45 remote sensing dataset is a large-scale public dataset published by the Northwestern Polytechnical University for remote sensing scene classification. The aircraft categories are shown in the original image in Figure 6. Then, we propose the imaging degradation by the original image and turbulence screen individually according to Equations (1) and (2). Finally, the degraded and the corresponding original images were obtained as inputs and labels in the dataset, respectively.

The whole dataset is divided into a training set, a validation set, and a test set in a ratio of 8:1:1. The network input image size was 256 × 256, and its label size was the same as its size. We used the L1 loss as the loss function.

$$Loss=\frac{1}{N}{\displaystyle \sum _{i=1}^N\left|y_i-{\widehat{y}}_i\right|}$$

(6)

where ${\widehat{y}}_i$ represents the output image, $y_i$ represents any image in the input image, and N represents the number of images in the batch.

3. Results and Discussion

3.1. The Test of Image Restoration in Simulation

In this section, we test the effectiveness and robustness of DeturNet using simulations. The training took approximately 3 h, with a total of 300 epochs. The initial learning rate was 1 × 10⁻⁴. The learning rate decay mode is cosine decay with a minimum learning rate of 1 × 10⁻⁶. The batch size is 16. The optimization algorithm was Adam, and the weight decay was set to 2 × 10⁻⁶.

The training platform was Pytorch 1.8.0, based on Python 3.7.11 in Sichuan, China. This is used to implement the network. An Intel Xeon(R) CPU (2.5 KHz) and NIVIDIA GeForce GTX 3090 GPU were used for the training and testing phases. The loss curve for the DeturNet training process is shown in Figure 7.

The restoration results are shown in Figure 8. The proposed results show that our method can effectively reduce the turbulence of an image. We also compared our method with other traditional and deep learning image recovery methods, such as the blind recovery method proposed by Jin et al. [46], the deep learning algorithms U-Net [42], and DeepRFT [47]. All the methods were retrained to achieve optimal recovery. The effects of the image restoration are shown in Figure 8. In order to evaluate the image restoration effectively, the results were analyzed using both subjective and objective methods. From a visual point of view, DeturNet has a better effect on turbulence image recovery, and the result is closer to the original image than those of the other methods. The other methods still have some degree of ambiguity in the edge information of the image target. Meanwhile, two evaluation functions, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), are proposed to perform an objective evaluation.

$$PSNR=10\cdot {\log }_{10}\left(\frac{MaxValu{e}^2}{\frac{1}{M\times N}{\displaystyle \sum _{i=0}^{M-1}{\displaystyle \sum _{j=0}^{N-1}{(X(i,j)-Y(i,j))}^2}}}\right)$$

(7)

$$SSIM(X,Y)=l(X,Y)\cdot c(X,Y)\cdot s(X,Y)$$

(8)

where X denotes the pixel value of the reference image, Y denotes the pixel value of the evaluated image, and (i, j) denotes the pixel coordinates. MaxValue is the maximum value of the color grey scale in an image, which is usually 255 on a uniform statement. l, c, and s represent the equations for brightness, contrast, and structure, respectively [48]. These two evaluation functions show different degradations of the images. PSNR usually shows the ratio between the maximum energy of the signal and the energy of noise, which affects the fidelity of its representation. SSIM usually shows the similarity between two images. A higher PSNR and SSIM closer to 1 often mean a better imaging quality [49,50,51].

In the outdoor experiments, the PSNR and SSIM evaluation criteria were not applicable due to the lack of corresponding labels. Therefore, we utilized three additional unreferenced evaluation criteria, namely variance, information entropy, and average gradient ($AG$), to objectively assess the recovery quality. The calculations for these criteria are as follows:

$$Variance=\frac{1}{M\cdot N}{{\displaystyle \sum _i{\displaystyle \sum _j(f(i,j)-\mu )}}}^2,\mu =\frac{1}{M\cdot N}{\displaystyle \sum _i{\displaystyle \sum _{j}f(i,j)}}$$

(9)

$$Entropy=-{\displaystyle \sum _{k=0}^{n}P(k){\log }_{2}P(k)}$$

(10)

$$AG=\frac{1}{M\cdot N}{\displaystyle \sum _i{\displaystyle \sum _j\sqrt{{\frac{{\left(\frac{\partial f}{\partial x}\right)}^2+\left(\frac{\partial f}{\partial x}\right)}{2}}^2}}}$$

(11)

where $M\cdot N$ represents the image size, $f(i,j)$ represents the gray value of the pixel $(i,j)$, and P(k) represents the ratio of the number of pixels with a gray value of k to the total number of pixels. $\frac{\partial f}{\partial x},\frac{\partial f}{\partial y}$ represents the gradients in the x and y directions, respectively.

In Figure 8, we can see that the evaluation index under the restored image of the DeturNet is the highest among all the methods in all four images. The average results of the two evaluation indices in the test sets are also shown in

Table 1. Comparison of the test sets.

	Blurred	Jin et al. [46]	U-Net	DeepRFT	DeturNet
PSNR(dB)	18.85	18.31	21.48	21.40	22.01
PSNR_std	2.40	2.19	2.37	2.60	3.20
SSIM	0.6745	0.6625	0.7304	0.7416	0.7644
SSIM_std	0.0848	0.0870	0.0934	0.0886	0.0833

, where PSNR_std represents the variance of the PSNR of the test set. For the test datasets, the recovery results show that DeturNet has an average improvement of 3.16 (16.8%) in the PSNR and 0.0899 (13.3%) in the SSIM, which is the highest among the four methods. Thus, our method performed better on all test sets and showed the superiority of our proposed method. We also compared our method with the other 4 methods in terms of time, as shown in

Table 2. Comparison of the average time consumed by methods to recover images.

Jin et al. [46]	U-Net	DeepRFT	DeturNet
113.38 s	48.87 ms	139.97 ms	47.16 ms

. The results show that our method has a better real-time capability compared to other methods. With the development of computer computing power, the proposed method has the potential to perform real-time imaging reconstruction.

3.2. Ablation

To further validate the important role of each module in DeturNet, this paper does further ablation experiments. It should be noted that the ablation experiments are not to verify the superior performance of a particular module but rather to demonstrate that DeturNet has good de-turbulence capability with the joint action of each module. The results of the ablation experiments are shown in

Table 3. Results of ablation experiments.

Method	SAM	CSFF	GAM	PSNR(dB)	SSIM
DeturNet	-	√	√	21.51	0.7614
	√	-	√	21.65	0.7597
	√	√	-	21.58	0.7576
	√	√	√	22.01	0.7644

. The selection of the hyperparameters is kept in line with the previously mentioned ones. It can be seen that the condition of removing a certain module leads to a decrease in the performance of DeturNet, and the de-turbulence ability of DeturNet is reduced.

3.3. The Robustness of DeturNet on Different Noises

To test the robustness of the network on different noises, four Gaussian-type noises with means of 0 and variances of 0.01, 0.03, 0.05, and 0.08 are added to the test set, and we feed it directly into our trained network. The test images and reconstructed images are shown in Figure 9. The results for the entire test set are shown in

Table 4. Comparison of the Gaussian test sets.

		Var = 0	Var = 0.01	Var = 0.03	Var = 0.05	Var = 0.08
Noise test sets	PSNR(dB)	18.85	18.83	18.77	18.69	18.55
	SSIM	0.6745	0.6654	0.6380	0.6064	0.5611
Recovery results	PSNR(dB)	22.01	21.86	21.50	21.30	20.89
	SSIM	0.7644	0.7460	0.7019	0.6603	0.6044

.

As we can see, the increase in noise affects both the blurred and restored images in Figure 9. When the noise is small, the reconstructed image is less affected by noise. When the noise increases, the reconstructed image shows a ringing effect; however, it still preserves the most detailed information. The target could still be clearly recognized. From Table 4, we can see that the PSNR and the SSIM are in a uniform decline, which indicates that the capability of our algorithm decreases slowly with an increase in noise, but the overall reconstruction capability is still good.

3.4. The Robustness of DeturNet on Different Turbulences

To test the robustness of our network under different turbulences, different turbulence intensities (D/r₀ = 5 and D/r₀ = 10) are proposed in this paper. So, we re-stimulated the datasets with their corresponding intensities and retrained them. The results are shown in Figure 10. The degraded image shows that a stronger turbulence intensity causes a more severely blurred image, which makes the recovery of turbulent degraded images extremely challenging.

The test results in Figure 10 show that, although the image is completely blurred by the turbulence of D/r₀ = 10, the recovery results of DeturNet can still reconstruct the target information, such as the aircraft outline and building edges. However, some high-frequency information is lost, such as the edge details in the recovered results, when the turbulence intensity increases further.

Table 5. Comparison of the Gaussian test sets.

	D/r0 = 5		D/r0 = 8		D/r0 = 10
	Blurred	DeturNet	Blurred	DeturNet	Blurred	DeturNet
PSNR(dB)	19.10	22.72	18.85	22.01	18.37	21.20
SSIM	0.6819	0.7797	0.6745	0.7644	0.6431	0.7302

shows that the reconstruction results are valid for the different turbulence intensity tests.

3.5. Laboratory Experiment Results and Discussions

In this section, we build a laboratory experimental imaging system to verify the performance and robustness of our network in practice. The laboratory connects the simulation and the outdoor experiments. Compared with the simulation dataset, it provides a more realistic atmospheric turbulence situation. Compared with outdoor experiments, it offers a dataset that is not affected by turbulence as the basis of the training dataset. As shown in Figure 11, LED and Digital Micromirror Devices (DMD) are used to generate the dynamic targets loaded by aircraft images of the NWPU-RESISC45 dataset, where the LED provides a stable, broad-spectrum light source, and DMD projects the loaded targets rapidly. A turbulent screen was used to generate the stochastic atmospheric turbulence, which had the same intensity as the simulation situations.

Through the experimental process, a dataset of 700 turbulence-blurred images is obtained and divided into a training set, a validation set, and a test set in the ratio of 8:1:1. The subjectivity of the recovered images and the two evaluation criteria show that our proposed method has a better reconstruction ability than the other methods. The partial recovery result images are shown in Figure 12. The average results for the test set are shown in

Table 6. Comparison of the test sets in experiment.

	Blurred	Jin et al. [46]	U-Net	DeepRFT	DeturNet
PSNR	18.73	18.70	24.19	27.10	27.69
PSNR_std	2.18	2.19	3.00	3.25	3.25
SSIM	0.5615	0.5573	0.6970	0.7719	0.7863
SSIM_std	0.1030	0.1058	0.0891	0.0710	0.0681

. Both the simulation and experimental results show that the recovery method based on DeturNet has better effects and robustness for turbulently degraded image restoration, which means that the proposed method has the potential to be used in image restoration for astronomical observation, remote sensing observation, and traffic detection.

3.6. Outdoor Experiment Results and Discussions

In order to further validate the effectiveness of our method, we collected real turbulence degradation pictures in the natural environment in the outside world. The experimental scene is shown in Figure 13a. In this case, the telescope focal length was 1250 mm, the object distance of the target was about 200 m, the CCD camera pixel size was 5.5 µm, the CCD camera exposure time was 3 ms, the acquisition time was 4 pm, the maximum temperature of the day was 33 °C and the minimum temperature was 24 °C. One of the acquired images is shown in Figure 13b.

In the experiment, we captured images of a toy car and calibration targets. We selected some regions of interest and tested the recoverability and generalization effect directly using DeturNet trained on a laboratory experimental dataset. The tested DeturNet-based single-frame recovery results are shown in Figure 14, from top to bottom, for scenarios 1–4. We observed the reconstructed images using both the subjective and objective methods. In a subjective evaluation, DeturNet exhibited a good recovery effect. The target contour edge information in the image was better recovered, and the boundary was more visible. Meanwhile, the variances of DeturNet restoration results were much larger than those of the blurred images and the results of the comparison methods.

Two objective evaluation criteria, information entropy and average gradient (AG), were used in this paper, as shown in

Table 7. Comparison of the test scenes.

		Blurred	Jin et al. [46]	U-Net	DeepRFT	DeturNet
Scene1	Entropy	6.08	6.13	6.95	6.76	6.80
	AG	1.78	1.66	2.84	3.73	2.86
Scene2	Entropy	6.55	5.91	7.27	7.54	7.32
	AG	3.55	3.73	6.38	7.67	6.75
Scene3	Entropy	7.31	7.34	7.67	7.73	7.64
	AG	2.87	3.26	5.21	5.44	5.71
Scene4	Entropy	6.85	6.70	6.87	7.42	6.86
	AG	2.59	2.67	4.36	4.69	3.78

. In the outdoor experiments, although the DeturNet was trained using the laboratory dataset, we can see that the network has good recovery results. Thus, from the perspective of visual effect, histogram distribution, and the size of the variance, entropy, and average gradient, the results show that we can recover the image scene from completely uninvolved training to a certain extent. The variance and entropy can reflect the amount of information in the image; the larger the value, the richer the information in the image hierarchy. The average gradient reflects the ability of an image to express the contrast in minute detail. In general, a larger average gradient often means the image is sharper. Since the original data are unknown, these evaluation functions have some limitations. For example, as we can see in the first row of images, even though the U-Net has the highest value of variance, the edges of the toy car in its recovered images are obviously distorted, while the edges of the toy car reconstructed using DeturNet are more realistic and sharper. Through subjective judgments supplemented by objective metrics, it can be seen that DeturNet outperforms other deep learning algorithms in terms of generalization ability. Meanwhile, the proposed results also show that the generalization ability can be improved with uncorrelated datasets when performing sufficiently fitted experiments, and we will continue our research on this issue.

4. Conclusions

In this paper, we propose a single-frame deep learning method called DeturNet for atmospheric turbulent image reconstruction. Compared with other deep learning methods, DeturNet has a deeper network level with a structure that is more consistent with turbulent characteristics. We verified the effectiveness of the DeturNet through simulations and experiments. Simulation and laboratory experimental results showed that the DeturNet has a better reconstruction effect and anti-noise ability compared to other methods. The outdoor experimental results show that the DeturNet has a good generalization ability after training with the lab-training dataset. These results show that DeturNet provides good recovery results for blind turbulence image reconstruction. When we combine the other advantages of deep learning, such as low cost and high speed, DeturNet can become an effective alternative method for the application of turbulent image reconstruction.