Cloud Contaminated Multispectral Remote Sensing Image Enhancement Algorithm Based on MobileNet

Abstract

Multispectral remote sensing images have shown unique advantages in many fields, including military and civilian use. Facing the difficulty in processing cloud contaminated remote sensing images, this paper proposes a multispectral remote sensing image enhancement algorithm. A model is constructed from the aspects of cloud detection and image enhancement. In the cloud detection stage, clouds are divided into thick clouds and thin clouds according to the cloud transmitability in multi-spectral images, and a multi-layer cloud detection model is established. From the perspective of traditional image processing, a bimodal pre-detection algorithm is constructed to achieve thick cloud extraction. From the perspective of deep learning, the MobileNet algorithm structure is improved to achieve thin cloud extraction. Faced with the problem of insufficient training samples, a self-supervised network is constructed to achieve training, so as to meet the requirements of high precision and high efficiency cloud detection under the condition of small samples. In the image enhancement stage, the area where the ground objects are located is determined first. Then, from the perspective of compressed sensing, the signal is analyzed from the perspective of time and frequency domains. Specifically, the inter-frame information of hyperspectral images is analyzed to construct a sparse representation model based on the principle of compressed sensing. Finally, image enhancement is achieved. The experimental comparison between our algorithm and other algorithms shows that the average Area Overlap Measure (AOM) of the proposed algorithm reaches 0.83 and the Average Gradient (AG) of the proposed algorithm reaches 12.7, which is better than the other seven algorithms by average AG 2.

1. Introduction

Remote sensing images are imaged by the response of specific spectral to ground objects. As remote sensing images generally contain multiple spectral, clouds will inevitably appear, especially since the cloud regions seriously affects the overall imaging effect [1]. The clouds have a certain transmittance in a specific spectral, and the transmittances are related to the thickness of the cloud. We regard the cloud with better transmittance as thin clouds, which can display their information from a spectrum, but the information is blurred. We consider thick clouds as difficult to transmittance clouds, where only a little information, or no information at all, is shown from the collected data. For this reason, extensive research has been conducted on clouds detection and image enhancement.

Generally, cloud detection algorithms are divided into traditional algorithms and deep learning algorithms. On one hand, traditional algorithms mainly analyze the difference between the cloud contaminated area and other areas from the perspective of the image. The representative algorithms mainly include the threshold method [2,3,4], which determines the threshold value using the statistical characteristics of image pixels to realize cloud area detection. This kind of algorithm has a fast operation speed. The filtering method [5,6,7,8,9] constructs a representative filtering model and realizes the segmentation of cloud areas according to the different response strengths of clouds and other areas. This kind of algorithm has a good detection effect on thick cloud areas, but it will encounter under segmentation or over segmentation phenomena for thin clouds. The feature method [10,11,12,13,14] constructs representative features from the perspective of space and spectrum to achieve cloud area extraction. However, this kind of algorithm analyzes the local area, which is easily affected by noise, and the segmentation is discontinuous in the process of cloud segmentation. The multi-source and multi-time phase fusion method [15,16,17,18] builds an information complementary model according to the different characteristics of different equipment, or builds a model based on the correlation of multiple acquisitions at the same location. This kind of method can better realize cloud detection, but it increases the cost of data acquisition and the difficulty of preprocessing, such as data registration. On the other hand, deep learning algorithms are mainly based on people’s cognitive processes to build a transmission response model to achieve intelligent identification. The representative algorithms are as follows. For the algorithms based on classification ideas [19,20,21,22], the response difference between the cloud and other areas is analyzed from the spectral information according to the collection characteristics of the labeled data in the early stage, and the principle of a large gap between classes is used to achieve cloud detection. The algorithms based on clustering ideas [23,24,25,26] analyze the common attributes between different clouds and use the principle of small intra-class differences to realize cloud detection. The above two types of algorithms have better generalization performance than traditional algorithms, but the detection accuracy still needs to be further improved. Multi-source feature response fusion [27,28] builds a deep model to extract representative features based on the differences in the response of different targets and different sources to achieve cloud detection. However, such methods increase hardware costs. The convolution response algorithm [29,30,31] constructs the mapping relationship between the input image and the detected image to realize cloud detection. This type of method has achieved good results, but also has some disadvantages, for example, the number of parameters is large, the training difficulty is related to the network structure, and the feature interpretability is low. Overall, the existing research shows that deep learning algorithms are superior to traditional ones. Traditional algorithms have good interpretability and fast computational speed. By comparison, deep learning algorithms show good detection performance and require a large amount of work in labeling and training in the early stage, but the feature interpretability is low.

At present, image enhancement algorithms are divided into image-based algorithms and imaging-based algorithms. On one hand, the image-based algorithms analyze the composition of image features to realize the enhancement of texture and color. The representative algorithms mainly include the texture enhancement algorithm [32,33,34], which constructs texture extraction operators to enhance the weakened texture information and has a better effect on images with richer textures. The color enhancement algorithm [35,36,37] builds a color mapping model to achieve image enhancement, which has a better effect on images with richer textures. Pixel statistics algorithm [38,39,40,41,42] builds a model according to the overall pixel distribution characteristics to achieve remapping. This type of method highlights the visual effect of the image, but the image will appear chromatic aberration. The multi-temporal and multi-source information fusion algorithm [43,44,45,46] realizes image enhancement by analyzing the relationship between different periods and signal sources. However, such an algorithm increases the hardware cost. Algorithms based on deep networks [47,48,49,50,51] build a mapping model based on the relationship between annotated images and actual images to achieve image enhancement. This type of algorithm has a strong dependence on data. On the other hand, imaging-based algorithms realize image enhancement by modeling the imaging sensor, environment, and other factors. The transfer function algorithm [52,53,54,55] constructs the transfer function of an imaging system and corrects the image’s imaging effect. This type of method has stricter requirements for the environment and sensor parameters. The specific condition enhancement algorithm [56,57,58] realizes image enhancement by establishing a model for the image analysis of a specific scene according to the actual application requirements. This type of algorithm is only applicable to limited scenes. In general, image-based algorithms and imaging-based algorithms build models from different angles, and both can achieve image enhancement. The universality of the above algorithms is still a key issue in the relevant research field.

Through the above analysis, in the aspect of remote sensing image enhancement of cloud pollution, cloud detection and remote sensing image enhancement are not combined, and the main problems faced are as follows. (1) The traditional image feature representation ability is limited, and the effect of cloud area extraction is not good. (2) The execution efficiency of the network structure is low, failing to meet the needs of practical applications. (3) The amount of cloud contaminated image data is small and the model generalization ability is poor. (4) The correlation between image imaging and image processing is poor, resulting in the need to improve the image enhancement effect.

From the aspects of cloud detection and remote sensing image enhancement, this research establishes a model and proposes a cloud contaminated multispectral remote sensing image enhancement algorithm based on MobileNet. (1) From the perspective of traditional features, a multimodal pre-detection algorithm is constructed. (2) From the perspective of deep learning, considering performance and practical application requirements, we improve MobileNet to fully extract target features. Based on the existing small sample data, a self-supervised training algorithm based on dilated convolution is constructed to learn features, and finally, cloud labeling is realized. (3) By analyzing the composition of the imaging signal, a multispectral fusion algorithm based on compressed sensing is proposed to achieve image enhancement.

The rest of this paper is organized as follows. In Section 2, the main framework of the algorithm proposed in this paper is introduced, and the multi-layer cloud detection and image enhancement algorithm based on compressed sensing is proposed. Section 3 introduces the database used in the experiment, and the main image data is presented. In Section 4, the effectiveness of the proposed algorithm is verified through a large number of experiments. Section 5 summarizes the innovations of this paper and possible follow-up work.

2. Methods

The specific process of the algorithm proposed in this paper is shown in Figure 1. Based on multimodal detection of thick cloud areas, MobileNet is improved to achieve fast and accurate thin cloud detection. The algorithm marks thick, cloudy, and cloudless regions because the hyperspectrum cannot penetrate the regions where there are thick cloud, and the marked regions are not processed. In the face of small cloud samples, the self-supervised training algorithm is used to enhance the robustness of the algorithm. On this basis, the multispectral fusion algorithm based on compressed sensing is used to enhance the image of the thin cloud and the cloudless regions, respectively, and the linear transformation is established to maintain the image consistency.

2.1. MobileNet

The MobileNet model is a lightweight model proposed by Google [59]. Currently, in response to the need for higher accuracy in image detection tasks, networks tend to be deeper and more complex, which inevitably leads to a large number of parameters and thus consumes a lot of computational resources. To balance the demands for time and performance, the MobileNet structure was accordingly proposed. On the basis of the traditional MobileNet structure, three typical structures have been derived.

MobileNet v1 is a lightweight neural network built on a pipelined structure using depthwise separable convolution with two hyperparameters, which can give developers greater freedom in selecting the suitable model [60]. This network structure has 28 layers, with depthwise convolution and pointwise convolution as two layers. The depthwise convolution is a convolution with a channel count of 1 and is responsible for the number of convolution kernels being the same as the input channel count at the output. The pointwise convolution kernel is 1 × 1 × M in size and aims to enrich the connections between channels for feature extraction.

MobileNet v2 focuses on the problem of linear bottlenecks in high dimensional space [61]. For example, the ReLU activation function, despite its effectiveness in increasing the nonlinear representation of features, is limited to high dimensional space, and using the ReLU function again will destroy the features if the dimensionality is reduced. Therefore, MobileNet v2 constructs a Linear Bottlenecks structure, where the activation function is no longer added for nonlinear transformation after the convolutional layer with dimensionality reduction has been performed, in order to cause as little information loss as possible. Instead, an inverse residual structure is introduced, and a bottleneck structure is proposed based on ResNet, which consists of two 1 × 1 convolutions and one 3 × 3 convolution. In MobileNet, the number of layers in depthwise convolution is the number of input channels. If, like the bottleneck in residual networks, compression is conducted first, followed by convolutional extraction, too few features can be obtained. Therefore, it takes an inverse approach, i.e., first up sample, convolution, and then down sample, to achieve the final feature extraction.

MobileNet v3 proposes a new activation function (h-swish), which is improved based on the swish activation function [62]. The swish function is unbounded, lower bounded, smooth, and non-monotonic, and it outperforms the ReLU function on deep models. This network structure allows the neural network to use global information to enhance useful information while suppressing useless information. The h-swish function achieves feature extraction by compressing, activating, and giving weights to different layers.

MobileNet adopts a top-down streamlined architecture, as shown in Figure 2, which mainly consists of a standard convolutional layer (Conv Std), a depthwise convolutional layer (Conv dw), and a linear correction unit (ReLU).

The network replaces traditional convolution with depthwise separable convolution, which greatly reduces the number of model parameters. Depthwise separable convolution can be divided into channel convolution and point convolution. The channel convolution adopts a 3 × 3 convolution kernel, and the point convolution adopts a 1 × 1 × M convolution kernel, where M corresponds to the number of channels in the previous layer to complete the weighted fusion of multi-channel images. The number of parameters is greatly reduced, as shown in Figure 3.

From the above analysis, it is clear that MobileNet is lightweight and is optimized at the convolutional level. However, for the multispectral remote sensing image data, this paper researches how to effectively focus on the cloud contaminated area, mark them, and then guide subsequent research on image enhancement.

2.2. Cloud Detection Algorithm Based on Improved MobileNet

In traditional feature analysis, there is a certain difference in the pixel distribution of the cloud on the image and other targets, but the traditional analysis algorithm cannot completely detect the cloud contaminated area. By comparison, the deep learning network is generally related to the input data and has many parameters, which cannot meet the requirements of fast and accurate calculation. To this end, we integrate traditional image features and deep features to construct a multi-layer cloud detection algorithm from cloud detection.

For single band thick cloud images, their histogram shows obvious double peaks. For less cloudy images, there is no obvious double peak structure, but a gentle extension in the histogram. The threshold method focuses on the trough of the histogram to realize cloud detection. Here, the discretization window is used to build the model and realize the filtering. For cloudy images, the OTSU algorithm [63] is used, and the threshold is set based on statistical principles to maximize the interclass variance. For the satellite image of ground objects, its histogram can be regarded as a multimodal state composed of multiple distributions. The fitting process is as follows:

$$g\left(x\right)={\displaystyle \sum _{i=1}^M{\alpha }_{j}P\left(x_i|{\phi }_j\right)}$$

(1)

$$P\left(x_i|{\phi }_j\right)=\frac{1}{\sqrt{2\pi {\sigma }_j^2}}\exp \left[-\frac{{\left(x_i-{\mu }_j\right)}^2}{2{\sigma }_j^2}\right]$$

(2)

where g(x) is the histogram fitting curve. M is the number of Gaussian models, which can be calculated from the Bayesian Information Criterion, x_i is the gray value of pixel I, P(x_i|φ_j) is expressed as the j th Gaussian distribution function, α is the weight, µ is the mean, and σ is the standard deviation.

We use the EM algorithm to estimate the parameters of the Gaussian mixture model. Specific steps are as follows. Firstly, set the initial parameters of M, α, µ, σ, and calculate the posterior probability:

$${\beta }_{ij}=\frac{{\alpha }_{j}P\left(x_i|{\phi }_j\right)}{{\displaystyle \sum _{i=1}^{M}P\left(x_i|{\phi }_j\right)}}, i\in \left[1,N\right], j\in \left[1,M\right]$$

(3)

Update the weights to get the standard deviation matrix C:

$$\left\{\begin{array}{l}{\alpha }_j={\displaystyle \sum _{i=1}^N{\beta }_{ij}/N}\\ {\mu }_j={\displaystyle \sum _{i=1}^N{x}_i{\beta }_{ij}}/{\displaystyle \sum _{i=1}^N{\beta }_{ij}}\\ C_j={\displaystyle \sum _{i=1}^N{\beta }_{ij}{\left(x_i-{\mu }_j^T\right)}^2}/{\displaystyle \sum _{i=1}^N{\beta }_{ij}}\end{array}\right.$$

(4)

When all parameters converge, the final parameters are obtained. The characteristics of the Gaussian distribution are estimated, using µ+3σ as the dividing line.

For thin cloud images, traditional algorithms cannot effectively extract the cloud area. In practical applications, the balance between accuracy and efficiency should be comprehensively considered. We conduct research based on the lightweight deep feature network MobileNet. To further improve the feature extraction ability of MobileNet, we learn from the idea of ViT (Vision Transformer), which directly applies the transformer to image processing. When tested with the same dataset, as the network depth increases, not only does the performance of ViT get saturated, but the attention maps appear similar after a certain number of deep computations. To solve the problem caused by the self-attention layer, DeepViT replaces the self-attention module in ViT with a re-attention module, and thus its performance will continue to improve as the model depth increases. The DeepViT model structure is shown in Figure 4.

The similarity between attention maps between transformer blocks is relatively high, especially in the deep layers. However, the similarity of different attention maps from the same transformer block is very small. So, we propose to exchange the attention of each head:

$$RAttention\left(Q,K,V\right)=Norm\left({\theta }^T\left(Soft\max \left(\frac{Q{K}^T}{\sqrt{d}}\right)\right)\right)V$$

(5)

where Q and K are the multiples for generating the attention map, $\sqrt{d}$ is the scale factor for the depth of the network, and θ is the transformation matrix.

The advantages of RAttention are as follows. (1) It uses the interaction between different attention heads to collect complementary information between them, which can better improve the diversity of attention maps. (2) The program of RAttention is very simple and easy to implement.

Given the lightweight advantages of the MobileNet network model and the performance advantages of DeepViT, we design a lightweight convolutional neural network model MobileDViT to reduce the number of parameters while maintaining high accuracy.

There are a total of seven bottleneck layers in the MobileNet network. The size of the image after MobileNet processing is H × W × C. The output needs to be processed by convolution of n × n and 1 × 1, and an original image H × W × d will be obtained. Then, it is input into the transformer, and a 1 × 1 point convolution is performed on the output result to obtain a H × W × C convolution. At this time, the output needs to be connected to the original image, and a H × W × C convolution will be obtained. The image of H × W × 2C is finally converted into H × W × C through a convolution of n × n with the input channel enlarged by 2 times and the output channel unchanged. For the overall structure of the MobileDViT block, the output after each convolution operation needs to go through the BN layer and the nonlinear activation function ReLU. In addition, the step of the MobileDViT block is 1 in the entire network structure. Among them, the n × n convolution in the MobileDViT block is standard, and its structure is shown in Figure 5.

MobileDViT block is similar to convolution, using two standard n × n and 1 × 1 convolutions and a transformer with RAttention, and replacing local processing in convolution with deeper global processing. So, the MobileDViT block can be regarded as convolution. Besides, it has properties similar to Convolutional Neural Network (CNN) and ViT, which helps to achieve better results with fewer parameters and a simple training method.

The Softmax regression model is used in training, which has good classification performance. By calculating the attribution probability of the input feature vector corresponding to each target class, the target is classified into the class with the highest probability. The class label y output by the Softmax model is a k dimensional real vector, where k corresponds to the number of classes. Then the probability of the model outputting class j is expressed as:

$$p\left(y_i=j\left|x_i;\theta \right.\right)=\exp \left({\theta }_{j}T\left(x_i\right)\right)/{\displaystyle \sum _{l=1}^k\exp }\left({\theta }_{l}T\left(x_i\right)\right)$$

(6)

where θ is the parameter of the Softmax model. For the test sample x, the probability of sample x belonging to each class is estimated by:

$$h\left(x,\theta \right)=\left[\begin{array}{l}p\left(y_j=1\left|x;{\theta }_1\right.\right)\\ \dots \\ p\left(y_j=k\left|x;{\theta }_k\right.\right)\end{array}\right]=\frac{1}{Z}\left[\begin{array}{l}\exp \left({\theta }_{1}T\left(x\right)\right)\\ \dots \\ \exp \left({\theta }_{k}T\left(x\right)\right)\end{array}\right]$$

(7)

$$Z={\displaystyle \sum _{j=1}^k\exp \left({\theta }_{j}T\left(x\right)\right)}$$

(8)

where Z is the normalization factor. The likelihood estimate is used as the loss function:

$$J\left(w\right)=\left[{\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^{k}Q\left(y_j^i\right)\log M\left(w\right)}}\right]$$

(9)

$$M\left(w\right)=\exp \left(w_{j}T\left(x_i\right)\right)/{\displaystyle \sum _{l=1}^k\exp \left(w_{l}T\left(x_i\right)\right)}$$

(10)

where $Q\left(y_j^i\right)$ represents the class function, being 1 when y_i belongs to class j, and 0 otherwise. However, the solving process is complicated. A weight decay term is added to make it have a unique solution, and the corresponding loss function is:

$$J\left(w\right)=-\frac{1}{m}\left[{\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^{k}Q\left(y_j^i\right)\log M\left(w\right)}}\right]+\frac{\lambda }{2}{\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^k{\theta }_{ij}^2}}$$

(11)

where λ > 0, and J is a convex function. The partial derivative of θ is obtained, and the gradient descent method is used to obtain the global optimal solution of θ. Based on this, a locally non-overlapping sliding window is constructed to search for pixels to estimate the concentration of the cloud contaminated area:

$$h\left(x,y\right)=\underset{\left(x^{\prime },y^{\prime }\right)\in W}{\min }\left(I\left(x^{\prime },y^{\prime }\right)\right)$$

(12)

where W is a window with a size of w × w and centered at (x,y). It is verified by the edge detection algorithm that for the blurred image, the local haze thickness map (HTM) is constant or changes little. Clouds are often concentrated in the low frequency part. To reduce the noise interference at the edge, the following function is constructed:

$$\begin{array}{l}H\left(x,y\right)=\min \left(h_i\left(x,y\right),h_m\right)\\ h_m=\frac{1}{n}{\displaystyle \sum _{\left(x,y\right)\in \mathsf{\Omega }}{h}_i\left(x,y\right)}\end{array}$$

(13)

After obtaining the cloud distribution map, it is necessary to restore the original image size and perform interpolation operations. However, this method is susceptible to noise interference and has a large gradient response at the edge, and the sliding windows do not overlap, which leads to agglomeration. Here, Gaussian curvature filtering is used to achieve noise suppression and edge smoothing. Curvature filtering is a method that minimizes the relevant curvature from the perspective of differential geometry to minimize the regularization term. The curvature is calculated by an implicit solution based on the continuity of differential geometry. Based on the discreteness of the data, the gradient and edge details of the image are maintained. Then, the target area covered by thin clouds is enhanced and the cloud contaminated area is detected.

2.3. Self-Supervised Training Algorithm

Due to the limited number of samples, a transfer algorithm needs to be used to improve the robustness of the algorithm. We adopt a transfer algorithm here, which mainly consists of a feature extractor and a classifier.

Feature Extractor. The residual attention dilated network is designed as a feature extractor, the dilated branch enables the network to extract features of different sizes, and the residual attention can enhance the model’s attention to important features.

Classifier. The fully connected layer and the Softmax layer are used as the class classifier. The fully connected layer processes the output of the feature extractor. The Softmax output is used for normalization to obtain the probability that the input data belongs to each class. To alleviate the over fitting phenomenon in the few-shot learning algorithm, the cross entropy loss function, with the addition of the L₂ regularization term, is used as the objective function of the network model.

Traditional convolutional networks use fixed size convolution kernels, which can only mine fixed size features and lack the attention of important features. To this end, we construct a residual attention dilated network. Firstly, based on the traditional convolutional network, the convolution process of expanding the sampling interval of the convolution kernel is adopted to expand the network’s receptive field, and features of different sizes can be extracted.

Using the Resnet12 network as a feature extractor and adding dilated convolution branches to it can not only expand the receptive field of the network but also extract features of different sizes. The commonly used methods to increase the receptive field of the convolutional network model are increasing the size of the convolution kernel or increasing the adjacent interval of the convolution kernel. A fixed size convolution kernel can only obtain the features of a fixed size, and an excessively large size convolution kernel cannot capture local delicate features. Therefore, this paper expands the field of view by increasing the convolution interval.

Based on the traditional Resnet model structure, a dilated convolution branch is added to the block of the feature extractor, and a new residual module is proposed.

The output X_i of each block in Figure 6 is expressed as:

$$X_i=Cov\left(X_{i-1}\right)+X_{i-1}$$

(14)

$$X_i=P\left(\max \left(X_i^{left},X_i^{right}\right)\right)$$

(15)

where P represents max pooling, X_i−1 represents the output of the previous block, $X_i^{left}$ represents the output of the left branch of the block, and $X_i^{right}$ represents the output of the right branch of the block. The update process is:

$$\left\{\begin{array}{l}{X}_i^{left}=Cov\left(X_{i-1}\right)+X_{i-1}\\ X_i^{right}=Co{v}_{A2}\left(X_{i-1}\right)+X_{i-1}\end{array}\right.$$

(16)

where the dilated convolution branch can capture the features of larger size, and each branch independently updates its scaling and shifting parameters to enhance model adaptability. To enhance the model’s attention to important features, the ability of the model to extract data is improved. An attention mechanism is introduced into each block, as shown in Figure 7.

The corresponding attention vector is expressed by:

$$M_i\left(X_{i-1}\right)=D\left(X_{i-1}\right)+\left(X_{i-1}\right)\ast U\left(D\left(D\left(X_{i-1}\right)\right)\right)$$

(17)

where D(.) is the subsampling using max pooling, and U(.) is the upsampling by bilinear interpolation. Then the output of each block is:

$$X_i=X_i+X_i\ast M_i\left(X_{i-1}\right)$$

(18)

Max pooling is not modified by back propagation to obtain scale invariant features. The characteristics of the image content are fully expressed. Finally, a convolutional network is obtained, which improves the receptive field of the model, extracts the features of different sizes, and enhances the model’s attention to important features. Thus, the model’s ability to extract data features is improved.

In the process of obtaining the basic network in the pre-training stage of transfer learning as shown in Figure 8, if the dataset used for training is very different from the new type of dataset, the generalization ability of the model is limited. We propose a pre-training algorithm based on self-supervision. By rotating the image data and establishing the image structure information label, the structure information of the image itself is fully mined, which increases the supervision information in the training task and enhances the generalization ability of the model.

The algorithm process is divided into two stages: pre-training and meta-learning. Self-supervision mainly uses auxiliary tasks to mine its own supervision information from large scale unsupervised data. The network is trained with this constructed supervision information.

A rotation-based self-supervised pre-training algorithm is used to rotate each batch of images, and the rotation angle is used as supervision information to establish labels to obtain more supervision information. A rotation angle classifier is built to train the model. In this stage, the feature extractor φ, the traditional class classifier θ, and the rotation angle classifier θ_s for the self-supervised auxiliary task are randomly initialized and optimized by gradient descent:

$$\left[\phi ,\theta ,{\theta }_s\right]=\left[\phi ,\theta ,{\theta }_s\right]-\alpha \nabla L_D\left[\phi ,\theta ,{\theta }_s\right]$$

(19)

$$L_D=\lambda L_{class}+\left(1-\lambda \right)L_{self}$$

(20)

where L_D is the loss function of cross entropy, and α is the learning rate.

2.4. Multispectral Fusion Algorithm Based on Compressed Sensing

Through the introduction of the above two sections, the detection of thin cloud area and thick cloud area under the condition of a small sample has been realized. We will focus on non-cloud and thin cloud areas to highlight features. Since the information contained in different spectral segments is different, to fully exploit the complementarity between the images, clearer images need to be obtained. However, the mainstream contourlet transformation has the following shortcomings. (1) The model does not have translation invariance. (2) The frequency aliasing effect will appear during the sampling process. (3) Insufficient orientation information exists in extracting rectangular images. To solve this problem, the contourlet transform is improved in this research. Specifically, we retain the advantages of multi-scale transformation, eliminate the subsampling step of the two layer transformation in the contourlet transform, and obtain translation invariance to avoid the frequency aliasing phenomenon of the fused image. A signal is considered sparsely representable if it can be linearly represented on a sparse basis. Multispectral images can be regarded as sparse images. For this reason, a fusion algorithm based on the principle of compressed sensing is constructed, as shown in Figure 9.

The multispectral image sequence {M} is decomposed by nonsubsampled contourlet transform (NSCT) to obtain high frequency sub band images {H₁, H₂, … H_s} and low frequency sub band images {L₁, L₂, … L_s}.

For high frequency sub band images, the average gradient $\left\{g_1,g_2,\dots g_s\right\}$ is calculated first. Then, the average gradient value G_F is obtained by using the CS based high frequency fusion principle. The high frequency fusion coefficient H_F is obtained according to the orthogonal matching tracking algorithm.

For low frequency sub band images, we first perform sparse dictionary learning, build a complete sparse dictionary D, implement sparse coding, and obtain sparse coefficients $\left\{{\alpha }_1^i,{\alpha }_2^i,\dots {\alpha }_s^i\right\}$. Then, the low frequency fusion coefficient L_F is obtained according to the low frequency sparse fusion rules based on dictionary learning.

The image is obtained by fusing high frequency and low frequency information. Since NSCT contains two transforms, in order to give full play to the characteristics of the two transforms, we adopt a two-layer decomposition structure, as shown in Figure 10. The first layer uses a non-subsampled pyramid filter bank (NSPFB) to obtain the low frequency sub band image I_L and the high frequency sub band image I_H. The second layer uses a non-subsampled directional filter bank (NSDFB) to decompose the I_H to obtain the corresponding high frequency sub band images I_Ha and I_Hb. Using NSPFB to decompose I_L, the corresponding low frequency sub band images I_La and I_Lb are obtained.

Due to insufficient information in the images of different infrared bands, if the maximum fusion rule or the weighted average fusion rule is adopted, the image will contain a lot of noise, and the display effect will be poor. Since the sparse dictionary can linearly represent the image’s sparse identification number, useful information can be effectively selected from it.

According to {H₁, H₂, … H_s} and {L₁, L₂, … L_s} obtained by NSCT, the image is divided into blocks using the sliding window technique, and the step size is set to s pixels to obtain q × q image blocks. ${\left\{P_L^i\right\}}_{i=1}^T$ denotes the T-th block of L with ${\left\{P_L^i\right\}}_{i=1}^T$. We convert $\left\{P_1^i,P_2^i,\dots P_s^i\right\}$ to a column vector as $\left\{V_1^i,V_2^i,\dots V_s^i\right\}$, and normalize it into:

$${\overline{V}}_m^i=V_m^i-{\overline{v}}_m^{i}C$$

(21)

where C is an n × 1 dimensional identity matrix, and ${\overline{v}}_m^i$ represents the mean of all elements. The sparsity coefficient is calculated according to the OMP (Orthogonal Matching Pursuit) algorithm:

$${\alpha }_s^i=\underset{\alpha }{\arg \min {‖\alpha ‖}_0}, s.t.‖{\overline{V}}_m^i-D\alpha ‖\le \xi {,}_{ }s\in \left\{1,2,\dots S\right\}$$

(22)

Use the L₁ norm to get the fusion coefficient:

$${\alpha }^i=\max \left({\alpha }_1^i,{\alpha }_2^i,\dots {\alpha }_S^i\right)$$

(23)

Then the corresponding sparse matrix is:

$$V_F^i=D{\alpha }^i+{\overline{v}}_F^{i}C$$

(24)

The low frequency fusion coefficient L_F is obtained by iterations according to the above steps.

The high frequency sub band image contains rich edge information and contains a large amount of structural information at different scales. For this reason, we design an algorithm to fully utilize the high frequency sub band image. Since the average gradient can reflect the grayscale changes on both sides of the boundary, it can be used to measure image clarity. This principle is used to build high frequency fusion rules. The average gradient value G of the high frequency sub band image is given by:

$$G=\frac{1}{\left(M-1\right)\left(N-1\right)}{\displaystyle \sum _{i=1}^{M-1}{\displaystyle \sum _{j=1}^{N-1}\sqrt{\frac{{\left(\partial f_x\right)}^2+{\left(\partial f_y\right)}^2}{2}}}}$$

(25)

where $\partial f_x$ and $\partial f_y$ are the horizontal and vertical derivatives, respectively. The reconstructed high frequency coefficients are:

$$H_F=\max \left(G_1,G_2,\dots G_s\right)$$

(26)

The fused image is obtained by inverse NSCT.

We normalized the regions separately for thin and cloudless masks, and Input NSCT for enhancement separately and then, the algorithm constructed a linear mapping relationship according to the difference between the pixel values of the thin cloud regions and the cloudless regions to achieve consistent image.

3. Dataset Description

We use 96 sets of data collected by Landsat 8. Landsat 8 consists of an Operational Land Imager (OLI) and a Thermal Infrared Sensor (TIRS) [64]. The similar resolution of the image is 7821 × 7951. A total of nine bands of remote sensing images are included in the dataset. We divide the dataset into three groups, as shown in Figure 11. Group 1 contains the images with large areas of cloud cover. Group 2 contains images of urban scenes, including typical landmarks such as houses and roads. Group 3 contains suburban scene images, including rivers, fields, and villages.

4. Experiment Analysis

Based on the above data, we use a Linux system, image server, and Python to load a deep learning framework for the experiment, using the same dataset to carry out the experiment. The ratio of test data to training data is 1:1.

The time consumption of the proposed image enhancement algorithm for Cloudy multispectral data, Urban multispectral data, and Suburban multispectral data is shown in

Table 1. The time consumption of image enhancement algorithm.

Algorithm	Cloudy	Urban	Suburban	Average
Time (s)	298	578	412	429

, the average of which is 429 s. As the Cloudy multispectral data contains a large number of cloud contaminated areas, the algorithm consumed a total of 298 s. The Urban multispectral data contains a wealth of ground object information and took a total of 578 s. The Suburban multispectral data, with a moderate amount of cloud and ground objects, took a total time of 412 s. We will subsequently verify the performance of the algorithm.

4.1. Performance of the Cloud Detection Algorithm

The analysis is conducted from the perspectives of traditional algorithms and deep learning algorithms. The histogram curve we extracted is shown as the blue line in Figure 12, and the red line is the multimodal fitting curve. It can be seen that the pixel distribution is multimodal. The area where the clouds are located is generally bright, and the ground objects are dark. Through the bimodal algorithm proposed in this paper, the pixel distribution was simulated, and the cloud contaminated area was initially extracted. Figure 12a shows a cloud free image. There is a lot of snow in the photographed area, showing a highlight area similar to the cloud. This area was extracted by the bimodal algorithm, and its attributes need to be further judged later. Figure 12b shows an image of an apogee thin cloud. The cloud covers most of the image, and the highlighted area concentrates many pixel values. The curve is unimodal, and there is a certain deviation in the selection of the threshold. Figure 12c is an apogee dotted cloud image. Because the cloud is high, it is dotted, the contrast of ground objects is low, and the brightness value is low. The curve shows a more obvious bimodal state, and the algorithm shows a good fitting effect. Figure 12d shows that the cloud area is blurred and there is no obvious texture area, which can be seen these clouds tend to gather and dissipate, so we should define them as changing clouds. Due to the rapid movement of the cloud, the blurred image presents a triple-bimodal form, which restricts the curve fitting effect. Figure 12e is an image of a thick cloud near the ground. The cloud texture features are clear, and the image shows an obvious bimodal state. Figure 12f is an image with both thick clouds and thin clouds taken near the ground. The proportion of ground objects in the image is small, and the double peaks are in a state of one large and one small. In summary, the algorithm proposed in this paper can realize the preliminary extraction of the cloud area. On this basis, it is necessary to further process the cloud contaminated area.

To verify the algorithm performance, this paper introduces AOM (Area Overlap Measure), AVM (Area Over-segmentation measure), AUM (Area Under-segmentation Measure), CM (Combination Measure) and ROC (Receiver Operating Characteristic) curves to evaluate the algorithm performance [65]:

$$\left\{\begin{array}{l}AOM=\frac{R_s\cap R_g}{R_s\cup R_g}\\ AVM=\frac{R_s-R_g}{R_s}\\ AUM=\frac{R_g-R_s}{R_g}\\ CM=\frac{1}{3}\left\{AOM+\left(1-AVM\right)+\left(1-AUM\right)\right\}\end{array}\right.$$

(27)

where R_g is the gold standard by manually annotated, and R_s is the algorithm segmentation result.

Table 2. Cloud detection results of database 1.

Algorithm	AOM	AVM	AUM	CM
CTF [4]	0.65	0.33	0.35	0.66
MS [12]	0.71	0.31	0.34	0.69
SVM [20]	0.75	0.26	0.27	0.74
ACNN [30]	0.76	0.21	0.25	0.77
The proposed algorithm	0.81	0.18	0.23	0.80

,

Table 3. Cloud detection results of database 2.

Algorithm	AOM	AVM	AUM	CM
CTF [4]	0.72	0.35	0.35	0.67
MS [12]	0.78	0.30	0.31	0.72
SVM [20]	0.80	0.26	0.22	0.77
ACNN [30]	0.82	0.17	0.19	0.82
The proposed algorithm	0.86	0.14	0.15	0.86

and

Table 4. Cloud detection results of database 3.

Algorithm	AOM	AVM	AUM	CM
CTF [4]	0.61	0.38	0.39	0.61
MS [12]	0.68	0.34	0.35	0.66
SVM [20]	0.76	0.26	0.31	0.73
ACNN [30]	0.80	0.21	0.23	0.79
The proposed algorithm	0.83	0.16	0.19	0.82

and show the average indicators of the running results of the comparison algorithms in the database. In general, Table 2 shows the results of cloud detection in database 1. In traditional algorithms, due to the large number of clouds included, the edge contrast is not strong, resulting in poor detection performance. Table 3 shows the results of cloud detection in database 2. Since the spectral response of the city is quite different from that of the cloud, the performance of the algorithm has been improved. Table 4 shows the results of cloud detection in database 3. Due to the constraints of the reflection of the river and the spectral response, the extraction effect is poor. Reference [4] proposed an algorithm for color and texture fusion (CTF), which has a better extraction effect for clouds with obvious colors and clear textures. Faced with complex and changeable scenes, the algorithm has limited adaptability. Reference [12] proposed a multi-scale (MS) algorithm, which considered the effect of different scales on cloud extraction. However, for images containing large areas of clouds, the feature extraction effect is poor. Reference [20] introduced a support vector machine (SVM) algorithm to convert cloud annotation into classification problems, which has a higher performance than traditional algorithms. Reference [30] proposed an attention mechanism based on CNN (ACNN) to focus on the area of interest and realize cloud annotation, but the algorithm requires a large amount of computation and the salient area has certain limitations. The algorithm proposed in this paper improves the MobileNet algorithm, as it comprehensively considers the requirements of accuracy and efficiency to achieve cloud annotation. In all, the proposed algorithm has the best performance. Figure 13 show the ROC curves by different algorithms. Figure 13a are the ROC curves of cloudless image. Cloudless images have high resolution. Figure 13b are the ROC curve of light clouds images. Light clouds images are blurred, so all algorithms are degraded. Figure 13c are the ROC curves of point clouds images. Point clouds images show that the areas where the clouds are blurred, and the area without cloud is clear. Figure 13d are the ROC curves of changing clouds images. Figure 13e are the ROC curves of near thick clouds. Due to the presence of some ground objects around the clouds, algorithms ROC curves are degraded. Figure 13f are the ROC curves of far thick clouds images. Due to the clouds being thick, and the distance is far away, the interference is small, and the algorithms present good effects. In general, the algorithm proposed in this paper achieves good results in different situations and can realize clouds detection.

4.2. Performance of the Training Algorithm

In order to verify the performance of the algorithm, indicators of Dice, Sen, and Spe are introduced to measure:

$$\left\{\begin{array}{l}Dice=\frac{2 \times TP}{2 \times TP + FP + FN}\\ Sen=\frac{TP}{TP + FN}\\ Spe=\frac{TP}{TP + FP}\end{array}\right.$$

(28)

where TP represents the number of pixels that are correctly predicted as positive samples, TN represents the number of pixels that are correctly predicted as negative samples, FP represents the number of pixels mispredicted as positive samples, and FN represents the number of pixels mispredicted as negative samples.

The core idea of the VGG-16 algorithm [66] is a small convolution kernel and multi-layer iteration. Due to the small receptive field, the training process has slow convergence and low accuracy. Inception v1 [67] separates cross channel correlation and spatial corrections, constructs a variety of different receptive field features, and sets many parameters. The Resnet network [68] adopts a multichannel architecture, which has strong interpretability, and its speed and accuracy are better than the above algorithms. MobileNet [60] uses depthwise convolution to improve the speed, and the spatial connection and cross channel are completely independent; while ensuring accuracy, the algorithm efficiency is greatly improved. It is proposed here based on the traditional algorithm; it introduces dilated convolution, expands the receptive field, and enhances the detection ability of large area clouds. The algorithm is robust and the training performance is further improved. The convergence curve is shown in Figure 14.

In order to visually demonstrate the performance of the proposed training algorithm, the cloud detection experiment was conducted under different conditions, with the results shown in Figure 15. Figure 15a is a substantially cloud-free image with scattered cloud points. Figure 15b is a thin clouds image. This area shows continuity, and the algorithm can detect thin clouds. Figure 15c is an image of the clouds presenting a dotted distribution. Figure 15d is an image containing moving clouds, showing a more obvious boundary. Figure 15e is a cloud-contained image taken near the ground, where the cloud has strong continuity and is obviously different from the ground objects. Figure 15f shows the interaction between thick clouds and thin clouds near the ground. The thick clouds are displayed as a large area of white, and the thin clouds are displayed as sporadic white. The difference between ground objects and clouds is obvious. It can be seen that the proposed algorithm can effectively detect the cloud contaminated area, and it has a good effect on the detection of no cloud, thin cloud, and thick cloud.

4.3. Evaluation of Image Enhancement Effect

The performance of the proposed algorithm is measured from the dimensions of subjective evaluation and objective evaluation. Experiments were conducted on cloudy, urban, and suburban images, as shown in

Table 5. Cloudy image enhancement effect.

Algorithm	AG	SD	SF	EI
LCM [56]	9.2	46.2	33.5	5.1
LEM [33]	9.5	47.5	34.5	5.3
SIH [39]	9.8	46.9	35.8	6.2
LGP [69]	10.5	49.8	35.4	6.4
CNN [30]	10.3	52.8	36.8	6.5
CVT [64]	10.9	51.3	37.2	6.8
MSF [45]	11.2	52.6	36.5	7.1
The proposed algorithm	12.3	56.1	38.6	7.3

,

Table 6. Urban cloud contaminated image enhancement effect.

Algorithm	AG	SD	SF	EI
LCM [56]	9.8	47.5	35.5	5.6
LEM [32]	10.1	49.8	36.1	5.9
SIH [39]	10.2	51.4	36.5	6.4
LGP [69]	11.6	52.6	37.1	6.5
CNN [30]	11.8	54.8	37.4	6.8
CVT [64]	12.1	55.9	38.6	7.3
MSF [45]	12.5	56.3	38.9	7.4
The proposed algorithm	13.1	58.2	40.1	7.6

and

Table 7. Suburban image enhancement effect.

Algorithm	AG	SD	SF	EI
LCM [56]	9.6	46.9	34.5	5.4
LEM [32]	9.8	49.1	34,3	5.7
SIH [39]	10.2	49.3	35.9	6.4
LGP [69]	11	52.1	36.4	6.5
CNN [30]	11.3	53.8	37.3	6.7
CVT [64]	11.4	53.5	38.1	7.1
MSF [45]	12.1	55.1	38.2	7.2
The proposed algorithm	12.7	57.7	39.1	7.4

.

Reference [56] established the luminosity and contrast model (LCM) from the perspective of color and reference [32] and established the local image enhancement model (LEM) from the perspective of texture to realize image enhancement. The whole image information is ignored. Reference [39] analyzed the overall distribution of the image, establishes a sub image histogram mode (SIH) to darken the region where the cloud is located, and enhanced the image value of other regions, so as to achieve the purpose of image enhancement. Reference [69] used the Laplacian-Gaussian pyramid (LGP) algorithm which has bright features, and the details are insufficient. Reference [30] used CNN to achieve the purpose of image enhancement. Reference [64] analyzed curvelet transform (CVT) and shows blurred textures. Reference [45] used multi-scale fusion (MSF) to image enhancement. As shown in Figure 16, the algorithm proposed in this paper can better maintain texture features and have a better display degree than other algorithms.

The following objective evaluation indicators are introduced for evaluation:

$$\{\begin{array}{l}AG=\frac{1}{\left(M-1\right)\left(N-1\right)}{\displaystyle \sum _{i=1}^{M-1}{\displaystyle \sum _{j=1}^{N-1}\sqrt{\frac{{\left(I_{i+1,j}-I_{i,j}\right)}^2+{\left(I_{i,j+1}-I_{i,j}\right)}^2}{2}}}}\\ SD=\frac{\sqrt{{\displaystyle \sum _{i=1}^M{\displaystyle \sum _{j=1}^N{\left(I_{i,j}-\mu \right)}^2}}}}{MN}\\ SF=\sqrt{\frac{{\displaystyle \sum _{i=1}^M{\displaystyle \sum _{j=2}^N{\left(I_{i,j}-I_{i,j-1}\right)}^2}}+{\displaystyle \sum _{i=2}^M{\displaystyle \sum _{j=1}^N{\left(I_{i,j}-I_{i-1,j}\right)}^2}}}{MN}}\\ EI=\frac{\sqrt{{\displaystyle \sum _{i=1}^M{\displaystyle \sum _{j=1}^N[s_x{(i,j)}^2+s_y{(i,j)}^2]}}}}{MN}\end{array}$$

(29)

where AG is the average gradient, M × N is the image resolution, SD is the standard deviation, µ is the mean, SF is the spatial frequency, EI is the edge intensity, and sx and sy are the convolution results of the Sobel operator.

5. Conclusions

In remote sensing imagery, it is inevitable that clouds will obscure ground objects, for which it is necessary to extract the cloud contaminated area for subsequent detection. If the image information acquired from a single band is limited, it is necessary to fuse multi-band image data to achieve image enhancement. Firstly, a multilevel cloud detection model is established based on the imaging characteristics of remote sensing images. For thick clouds, the bimodal model is constructed. For thin clouds, the RAttention transformer block is introduced into MobileNet to extract thin cloud areas, and then a cloud thickness metric model is constructed to label the penetrable thin clouds. Next, based on the self-supervised training framework, we can perform network learning under the condition of a small sample. In terms of image enhancement, a compression-based perception model is developed. The NSCT algorithm is then constructed by analyzing the information contained in each waveband image in the frequency domain to achieve image enhancement. Finally, experiments were conducted with four categories of databases: a cloudy area; a city with thin cloud; a city with thick cloud; and a suburb. For the proposed algorithm, the average AOM reached 0.83 and the average AG reached 12.7, showing the enhanced image quality compared with the original image quality.

The innovations of the proposed MobileNet based multispectral remote sensing image enhancement algorithm for cloud contamination can be summarized as follows:

(1) From the perspective of traditional features, we construct a pre-detection algorithm based on multiple peak states.

(2) From the perspective of deep learning, considering performance and practical application requirements, we improve MobileNet to fully extract the target features. Based on the existing small sample data, a self-supervised training algorithm based on dilated convolution is constructed to learn the features and finally achieve cloud annotation.

(3) By analyzing the signal composition of the obtained image, we propose a compression-aware multispectral fusion algorithm to achieve image enhancement.

Despite some progress in cloud contaminated multispectral remote sensing image enhancement algorithms, some issues still need to be addressed in the future:

(1) At present, we use multispectral means to mark cloud-contaminated areas, and it is a more important task to follow up with other technical means to recover cloud contaminated areas.

(2) Multispectral remote sensing data has certain limitations due to the limited operating time of the satellite payload. Therefore, mining the relationship between the available data to achieve image quality enhancement is a major task.

(3) Existing algorithms process data from ground stations, so it is of great significance to develop a stable and high-performance satellite side processing system.