Daytime Sea Fog Detection Based on a Two-Stage Neural Network

Abstract

Sea fog detection has received widespread attention because it plays a vital role in maritime activities. Due to the lack of sea observation data, meteorological satellites with high temporal and spatial resolution have become an essential means of sea fog detection. However, the performance is unsatisfactory because low clouds and sea fog are hard to distinguish on satellite images because they have similar spectral radiance characteristics. To address this difficulty, a new method based on a two-stage deep learning strategy was proposed to detect daytime sea fog in the Yellow Sea and Bohai Sea. We first utilized a fully connected network to separate the clear sky from sea fog and clouds. Then, a convolutional neural network was used to extract the differences between low clouds and sea fog on 16 Advanced Himawari Imager (AHI) observation bands. In addition, we built a Yellow and Bohai Sea Fog (YBSF) dataset by pixel-wise labelling AHI images into three categories (i.e., clear sky, cloud, and sea fog). Five comparable methods were used on the YBSF dataset to appraise the performance of our method. The vertical feature mask (VFM) generated by Cloud-Aerosol Lidar with Orthogonal Polarization (CALIOP) was also used to verify the detection accuracy. The experimental results demonstrate the effectiveness of the proposed method for sea fog detection.

1. Introduction

Compared with daytime fog detection over land, fewer observation stations are mounted at sea, therefore, it is a challenge to detect the sea fog with sparse observation data for further investigation. In recent decades, researchers have leveraged meteorological satellite data with high temporal and spatial resolution for large-scale sea fog detection [1]. The current mainstream strategy for sea fog detection is selecting the optimum threshold values based on the radiation property of sea fog. Hunt [2] theoretically proved that the specific emissivity of opaque water clouds such as fog and low clouds in the short-wave infrared bands was significantly lower than that in the long-wave infrared bands. Based on this assumption, Eyre et al. [3] adopted the brightness temperature difference (BTD) between the 3.7 μm and 11 μm bands of the NOAA polar-orbiting satellite to detect heavy fog at night. Ellrod [4] successfully applied the BTD method to the GOES geostationary satellite for long-term fog monitoring and analysis. Because the short-wave infrared channel received both the radiation of the target itself and the reflection of the solar radiation from the target during the daytime, the BTD method cannot be directly applied for daytime fog detection. Nevertheless, as more visible bands are available during the daytime, Bendix, et al. [5] utilized visible and near-infrared band data to conduct daytime fog detection with Moderate Resolution Imaging Spectroradiometer (MODIS) data.

In order to make full use of the multi-spectral satellite data, the spectral radiance characteristic difference of fog and other targets was analyzed and compared based on a large number of satellite observations [6,7]. Hao et al. [8] simulated the radiation properties of fog/clouds with the streamer radiative transfer model under different fog/cloud microphysical and observational conditions. The spectral radiance characteristics provide a reliable basis for the band selection of sea fog detection. Zhang et al. [9] discussed a variety of spectral indices designed for sea fog remote sensing detection including the normalized difference of snow index (NDSI) [10] and the normalized difference near-infrared water vapor index (NWVI) [11], and built a multi-source satellite-based full-time sea fog detection model.

Considering that the remote sensing data are easily affected by factors such as instrument difference, solar elevation angle variations, and seasonal changes, it is not suitable to adopt the same threshold values to carry on sea fog detection from the data observed by satellites in different regions and at different times. Cermak and Bendix [12] dynamically selected the threshold based on the histogram of radiance difference in a local area, which could overcome the above challenges to a certain extent. Drönner et al. [13] first divided the European area covered by the MSG satellite into uniform grids, then dynamically determined the low cloud and heavy fog separation threshold in each grid according to the histogram of the BTD, and used OpenCL parallel processing to monitor fog continuously. Kim et al. [14] proposed a decision tree (DT) approach for sea fog detection based on Geostationary Ocean Color Imager (GOCI) observations in combination with Advanced Himawari Imager (AHI) data onboard the Himawari-8 satellite, and reduced the sensitivity to the highly variable environment.

Although the dynamic threshold methods improve the accuracy of sea fog detection, it is difficult to determine the optimum threshold value based on the peaks and troughs of the spectral histogram, since it varies considerably. With the development of deep learning, convolutional neural networks (CNN) have widely been used in many fields such as image recognition [15] and target detection [16,17] and provide state-of-the-art performance. CNNs can learn discriminative texture features by a training dataset. In contrast, it is inefficient for traditional approaches to define and extract features of sea fog due to the complexity of natural texture and patterns. However, deep learning methods are rarely used in the research of sea fog detection because it has a high cost in terms of human resources for labeling the training datasets [18]. Only a few articles have described the application of deep learning in sea fog detection. Si et al. [19] extracted 33-dimensional combined features from the AHI data including the reflectance/brightness temperature of each band of the AHI data and the texture features extracted from the 0.64 μm band, and utilized a fully connected neural network for daytime sea fog recognition, achieving an accuracy of 82.63%. Jeon et al. [20] generated 120 images of each category with a size of 100 × 100 pixels from GOCI and used them to fine tune the pre-trained VGG19 [21] and ResNet50 [22]. Although the highest accuracy in the experiment reached 91.6%, the model seemed to lack sufficient robustness due to the small amount of training data for the image classification task. Zhou et al. [23] proposed a new sea fog detection dataset (SFDD) that labeled 1032 sea fog images recorded by the GOCI over the Yellow Sea and Bohai Sea, and designed a dual-branch sea fog detection network (DB-SFNet) to jointly extract discriminative features from both the visual and statistical domains.

To this end, in this paper, we constructed a daytime sea fog remote sensing dataset called YBSF (Yellow and Bohai Sea Fog). Compared with the GOCI data (eight channels, 0.41 μm~0.68 μm), the YBSF dataset based on Himawari-8 (16 channels, 0.47 μm~13.3 μm) can provide more spectral information for sea fog detection, and the YBSF dataset was also available for peer researchers. Two meteorological experts labeled and cross-validated the dataset, which classified each pixel in 24 AHI images into four categories: clear sky, land, cloud, and sea fog, which were assigned as the ground truth. Based on this dataset, a two-stage neural network was proposed for daytime sea fog detection. Unlike other sea fog detection methods based on machine learning that distinguish clear sky, cloud, and sea fog at the same time, or treat clear sky and cloud as the same category, the proposed method first regarded sea fog and cloud with similar spectral characteristics as one category and designed a multilayer perceptron (MLP) to distinguish clear sky and non-clear sky pixels. Then, the proposed CNN model was used to extract more discriminative spectral and texture features to separate sea fog from cloud. The proposed method was trained and tested based on the YBSF dataset, and was also validated using the vertical feature mask (VFM) generated by Cloud-Aerosol Lidar with Orthogonal Polarization (CALIOP) onboard the Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observation (CALIPSO) [1]. Experimental results on the YBSF dataset showed that the rich spectral and textural features extracted by the proposed neural network were more capable of separating cloud and fog than the comparative methods.

2. Data

2.1. Study Areas

In this paper, we selected the northern part of the Yellow Sea and Bohai Sea as the study area. Figure 1 shows a sea fog occurrence in that area on 18 March 2020, at 01:00 UTC. The Yellow–Bohai Sea area has experienced heavy fog events over a considerable fraction of the year [24]. Sea fog in this area is mostly from advection fog events, which are generated when the prevailing south wind brings warm and humid air to the cold northern waters during the sea fog season from March to August [25,26,27].

2.2. YBSF Dataset

Himawari-8 is the latest generation of the geostationary meteorological satellite funded by the Japan Meteorological Agency and developed by Mitsubishi Electric Corporation, which is equipped with the most advanced optical sensor AHI. The AHI consists of 16 observation bands covering visible, near-infrared, and infrared bands, with spatial resolution varying between 0.5 km and 2 km, and offers full disk observation data every 10 min.

Table 1. The Himawari-8/AHI specifications.

	Band	Wavelength (μm)	Spatial Resolution (km)
Visible	1	0.47	1
	2	0.51
	3	0.64	0.5
Near-infrared	4	0.86	1
	5	1.6	2
	6	2.3
Infrared	7	3.9
	8	6.2
	9	6.9
	10	7.3
	11	8.6
	12	9.6
	13	10.4
	14	11.2
	15	12.4
	16	13.3

shows the details of the 16 AHI bands along with the related wavelengths [28].

In this paper, we collected 24 sea fog events captured in 2020 from the study area to construct the YBSF dataset. Each pixel of the dataset was labeled as one of four categories, namely, ground, clear sky (i.e., the sea water pixels, which is not covered by cloud or fog), sea fog, and clouds, respectively, by two experienced meteorological experts, based on the spectral differences between different categories, and the manually labeled results were taken as the ground truth. Since the radius of fog droplets is close to the wavelength of the near-infrared channel, according to the Mie scattering theory, the reflectivity of this channel is greater than that of the visible channel, while the reflectivity of cloud droplets in the visible channel is greater than that in the near-infrared channel. Meanwhile, as the sea fog is close to the ground, its brightness temperature in the infrared channel is higher than that of the cloud. The 500 m spatial resolution of 0.64 μm data also provides rich texture information for cloud and sea fog discrimination. Thus, we synthesized color images with 0.64 μm, 1.6 μm, and 11.2 μm band data to help experts label the dataset more accurately.

For the proposed method, 80% of the YBSF dataset was used for training and the rest was used for testing. We randomly sampled 10,000 clear sky pixels and 10,000 non-clear sky pixels from the training set for the first-stage MLP model training. For the second-stage CNN model, we randomly sampled 50,000 cloud-centered image patches and sea fog-centered image patches, respectively. The training samples of the extraction procedure is illustrated in Figure 2. The selection of the image patch size will be discussed in Section 4.

3. Method

Detecting sea fog from satellite images remains challenging for traditional methods based on thresholds because remote sensing spectral data vary over time and space. Neural networks can extract high-level feature representation (i.e., texture) and are naturally considered to be a reasonable choice to address the difficulty of sea fog detection. Thus, we propose a new sea fog detection method using a two-stage neural network architecture, which integrates multi-spectral and spatial information of the AHI data. The first stage was dedicated to recognizing clear sky, and the second stage was committed to distinguishing between clouds and sea fog areas. Benefitting from the step-by-step method, we located the sea fog region with high accuracy. In this section, we initially sketched the components of the proposed neural network architecture and then elaborated on the details of each procedure. The overall flow chart of the proposed method is illustrated in Figure 3. Each 16-dimensional spectral pixel of the AHI image was first input to a fully connected MLP for clear sky pixel detection. Then, for each non-clear sky pixel, an image patch was extracted centered on this pixel as the input of the convolutional neural network to fully extract the spatial information for cloud and sea fog separation.

3.1. Pre-Processing

The satellite images need to be preprocessed into a suitable format before being fed to the neural network. Each original AHI data of the survey area were converted into a 16-band image with geographic latitude/longitude projection. The spatial resolution of each image was 0.005 degrees (~0.5 km), thus the size of the image was 2000 × 1600. After that, the land mask was applied to each image to obtain the region of interest (ROI), as shown in Figure 3. All pixels in the ROI were classified into three categories: clear sky, sea fog, or clouds, and the pixels of the land were excluded.

3.2. Distinguish Clear Sky from Clouds and Sea Fog

It is widely accepted that the spectral properties of the clear sky and sea water are very different from those of clouds. To obtain as much spectral information as possible, we employed a MLP architecture that took 16-band AHI pixels as the input and predicted whether the pixels were clear sky or not. The MLP consisted of two hidden layers with ReLU (rectified linear unit) as the activation function, as shown in Figure 4. The first hidden layer was fully connected to the original 16-dimensional spectral vector to extract the 4-dimensional combined spectral feature vector. Through the second hidden layer, the spectral features were continuously fused and reconstructed to obtain 3-dimensional deep spectral features. The output layer comprised two perceptions, which classifies the input pixel into two categories (i.e., clear sky and non-clear sky). After excluding the clear sky areas through MLP, the non-clear sky pixels of the ROI (consisting of both cloud area and sea fog area) will serve as input to the second stage to distinguish sea fog from clouds.

3.3. Distinguish Sea Fog from Clouds

Sea fog can be regarded as a special kind of cloud, which has certain similarities with clouds in spectral characteristics. Therefore, it is difficult to distinguish clouds and sea fog by spectral vector directly. The large-scale observation capability of satellite cloud images can obtain the spatial characteristics, which is an effective criterion to separate sea fog from clouds. To fully exploit the differences in texture and spatial structure of these two categories in the satellite images, we adopted a CNN-based architecture for sea fog detection, as shown in Figure 5. For each non-clear sky pixel, a 65 × 65 sized image patch with 16 bands centered at the pixel was cropped as the model input. The spectral and spatial features of the image patch were extracted at different scales through three convolutional layers, each followed by a max-pooling layer. The feature map was then flattened and fed into two fully connected layers to classify the center pixel of the image patch as sea fog or clouds. When training the model, random flip and rotation were also adopted for data augmentation.

4. Experimental Results and Analysis

In this section, the prediction performance of the proposed two-stage neural network is reported on the YBSF dataset. We conducted the experiments with the proposed algorithm and other state-of-the-art sea fog detection methods. We compared different methods with five evaluation metrics including false alarm ratio (FAR), probability of detection (POD), critical success index (CSI), percentage error (ERR), and Hanssen–Kuiper skill score (KSS). Definitions of the metrics are as follows:

$$FAR=\frac{FP}{TN+FP}$$

(1)

$$POD=\frac{TP}{TP+FN}$$

(2)

$$CSI=\frac{TP}{TP+FP+FN}$$

(3)

$$ERR=\frac{FN+FP}{TP+TN+FP+FN}$$

(4)

$$KSS=POD-FAR$$

(5)

where FP, FN, TP, and TN indicate false positives, false negatives, true positives, and true negatives.

Furthermore, to validate the effectiveness of the proposed method, we applied the CALIPSO VFM data to evaluate the sea fog detection accuracy, which will be discussed in Section 4.3.

4.1. Image Patch Size Analysis

In this section, the size of the image patch was investigated to assess its impact on the sea fog detection accuracy. The detection results with different image patch sizes showed that the sea fog detection accuracy was relatively high when the patch size was 65 × 65, as elaborated in

Table 2. Classification accuracy of the proposed method with different image patch sizes.

Size	FAR	POD	CSI	ERR	KSS
33	0.088	0.936	0.711	0.083	0.848
65	0.059	0.919	0.758	0.064	0.860
97	0.067	0.889	0.717	0.076	0.822
129	0.054	0.875	0.733	0.069	0.821

, the best value for each evaluation matric is in bold. The reason is that an appropriate image patch can capture more abundant texture information of the sea fog and thus enhance the discrimination level of the CNN for the central pixel of the image patch. However, when the image patch is too large, it might include objects with different categories, which causes confusion in the recognition of the central pixel.

4.2. Comparison with Other Methods

To validate the superior recognition performance of the proposed algorithm, in this section, we compare the results with five state-of-the-art sea fog detection methods including support vector machine (SVM), MLP, VGG16 [14], U-Net [29], and the threshold method [12], as shown in

Table 3. A comparison of the experimental evaluation metrics of different methods.

Method	FAR	POD	CSI	ERR	KSS
SVM	0.070	0.898	0.718	0.077	0.828
MLP	0.077	0.897	0.702	0.083	0.820
VGG16	0.056	0.847	0.705	0.077	0.791
Ours	0.059	0.919	0.758	0.064	0.860
U-Net	0.053	0.885	0.742	0.067	0.831
Threshold	0.107	0.942	0.679	0.097	0.835

, the best value for each evaluation matric is in bold. The SVM method trained an optimal hyperplane to separate sea fog using the 16-dimensional spectral training set. The MLP method used the first stage of our method to classify sea fog, cloud, and clear sky at the same time. VGG16 and U-Net distinguished sea fog from cloud by extracting hierarchical features from 3-band images (B03, B05, and B14) with multiple convolutional and pooling layers. The threshold method detects fog based on a chain complementary spectral and spatial property test. For the spectral tests, the following channels were used: 0.6 µm, 0.8 µm, 1.6 µm, 3.9 µm, 8.7 µm, 10.8 µm, and 12.0 µm. A threshold value of BTD at 10.8 µm and 3.0 µm was dynamically determined to remove cloud-free areas. The BTD at 12.0 µm and 8.7 µm was used as an indicator of the cloud phase, and radiances in the 3.9 µm channel were adopted for the small droplet proxy test. When applying this method to the AHI data, we chose the following channels with similar central wavelengths: B03 (0.64 µm), B04 (0.86 µm), B05 (1.6 µm), B07 (3.9 µm), B11 (8.6 µm), B14 (10.4 µm), and B15 (12.4 µm).

Based on the experimental evaluation metrics in Table 3, we can conclude that the machine learning-based methods generally achieved better results. The proposed two-stage network worked well on most of the evaluation indices. The threshold method performed the best on POD, and our method ranked second on this index. However, the FAR of the threshold method was much higher than that of other methods and its CSI score was the lowest among all methods, indicating that the threshold method is prone to misclassifying other categories as sea fog. U-Net achieved the lowest FAR, but its POD was also poor, which means that U-Net cannot effectively distinguish sea fog from clouds with similar characteristics. It should be noted that our method used the image patch as the input of CNN to classify the central pixel. Therefore, when the center is located at the junction of the sea fog and cloud cover area, it may bring about misclassification because it contains cloud and fog information at the same patch. Overall, our method achieved a good balance on the POD and FAR metrics, and the overall superiority of all metrics showed that our method achieved the best detection performance.

Figure 6 demonstrates the sea fog detection results on the test dataset. The detection error was mainly caused by the confusion of clouds and sea fog, especially when low clouds and sea fog both exist in the ROI. For the first case, the cirrus cloud over the sea area in the lower right corner areas was misclassified as sea fog by the comparative methods. The sea fog area in the second case was not affected by the cloud; all methods except for U-Net achieved good results. The U-Net method incorrectly segmented a small area inside the sea fog area as cloud. For the third case, the sea fog was occluded by clouds; all methods could only detect the uncovered fog area, and misclassified a small part of the cloud as fog, among which the U-Net method had the most serious misclassification. The threshold method misclassified some shallow water areas adjacent to the continent as cloud areas when detecting the fourth case. In the fifth case, due to the interference of low clouds, all methods found it difficult to effectively separate the low cloud areas in the northwest and southwest seas, and the accuracy of CNN and our method was relatively high. Compared with the results of all of other methods, the proposed method had better detection accuracy at the sea fog boundaries and better regional consistency. Generally, the proposed algorithm achieved almost the best classification results in various sea fog conditions in terms of the quantitative and qualitative assessment.

4.3. Validation with CALIPSO VFM

Although the AHI has the characteristics of high spatial resolution and sustainable observation, providing a new level of capability for detecting sea fog, it fails to offer the cloud base height parameter for discrimination between low-level stratus and fog. The problem is particularly evident in ocean areas, where no ground observations are available [30]. The CALIOP was jointly developed by NASA (National Aeronautics and Space Administration) and CNES (French Centre National d’Etudes Spatiales) and became operational in 2006. The CALIOP has two bands of 532 nm and 1064 nm, which can penetrate clouds and aerosols and obtain atmospheric vertical profile structure information [31]. The Level 2 products of CALIOP, known as CALIPSO VFM products, provide information on cloud and aerosol locations and types. Although the CALIPSO VFM can fill the gap with the capability of vertical profiling, it does not always meet the needs, since it follows its orbit linearly rather than in real-time. Therefore, in this section, the VFM products were used to validate the fog detection accuracy of the proposed algorithm.

Figure 7a shows a sea fog case observed by Himawari-8 on 8 June 2018 05:10 (UTC) with CALIPSO passes over (solid white line) the study area at nearly the same time (8 June 2018 05:09:30–05:11:42 UTC). The detection result of our method illustrates that the sea fog almost fully covered the Yellow Sea and Bohai Sea area. Sea fog pixels were detected along the A-B, C-D, and E-F line segments, as displayed in Figure 7b. The CALIPSO VFM data demonstrate that there were clouds connected to the sea level (i.e., sea fog, according to the definition of fogs [32]) along three footprint line segments A-B, C-D, and south E-F in Figure 7c as well as rough surface (i.e., sea fog, identified as sea fog by [33,34,35]) located in the north half of E-F. These low-level clouds touching the sea surface identified by VFM data corresponded reasonably with the sea fog detection results of the proposed method. The CALIPSO also observed high clouds located at approximately 7–12 km in these three segments, which proves that the proposed method can detect the sea fog covered by transparent cirrus clouds to a certain extent. The validation experiment proved that our method has promising application for the detection of sea fog.

5. Conclusions

In this paper, we built a daytime sea fog dataset called YBSF that consisted of a sequence of AHI images with ground truth labels. A two-stage neural network-based sea fog detection method was presented and tested on the proposed dataset. The experimental results show that, compared with the traditional machine learning methods or classical neural network architectures, the two-stage neural networks algorithm achieved the highest accuracy for sea fog detection. Furthermore, the proposed method had excellent advantages in digging out the data connections between different channels through convolution networks, so that it does not need a careful selection of channel data for discriminatory input features, which eases the workload of analyzing and selecting channel combinations for meteorologists. Many researchers believe that partial channel combination thresholds can separate clear sky, clouds, and sea fog. However, we conducted a series of experiments and concluded that full channel data, instead of partial channel combination data, as input of data-driven algorithms, can outperform in detecting sea fog tasks. Further validation on the CALIPSO VFM data also confirmed that our method is effective for sea fog detection.

It should be noted that from the perspective of meteorological satellites, sea fog is often obscured by clouds above it. The existing methods including the proposed method fail to efficiently extract the fog area under cloud cover. Geostationary meteorological satellites can obtain time-dependent changes in the occlusion of fog by clouds through continuous observation. How to effectively use temporal information to extract sea fog under cloud cover will be the next research effort.