Deep Learning-Based All-Sky Cloud Image Recognition

Abstract

Accurate cloud identification is crucial for understanding the rapid evolution of weather systems, improving the accuracy of short-term forecasts, and ensuring aviation safety. Compared with traditional cloud image recognition methods, deep learning technology has advantages such as automatic learning of complex features, high-precision recognition, and strong robustness in changing environments, providing more reliable and detailed cloud information. This study utilized 256 cloud image observation data points collected by an all-sky imager from 3 to 30 November 2023, at the Tunchang County Meteorological Bureau in Hainan Province (19°21′N, 110°06′ E). A Convolutional Neural Network (CNN) model was employed for cloud image recognition. The results show that in terms of cloud recognition, the constructed CNN model achieved an accuracy rate, recall rate, and ${\mathrm{F}}_1$ score of 100%, 91%, and 95%, respectively, for clear skies and stratus clouds, cumulus clouds, and cirrus clouds, with an average recognition accuracy rate of 95%. In terms of cloud cover detection, when comparing the Normalized Red Blue Ratio (NRBR) and K-Means clustering algorithm with the system’s built-in monitoring results, the NRBR method performed optimally in cloud region segmentation, with cloud cover estimates closer to the actual distribution. In summary, deep learning technology demonstrates higher accuracy and strong robustness in all-sky cloud image recognition.

1. Introduction

Clouds serve as a core regulatory factor in Earth’s energy balance [1], and their distribution, macroscopic morphological characteristics, and temporal evolution patterns exert a profound influence on global climate patterns [2]. Given that the cloud types defined by the World Meteorological Organization (WMO), such as cirrus and cumulus clouds, not only exhibit significant visual differences [3] but also play unique roles in atmospheric thermodynamics and radiation transport processes, achieving precise identification of cloud types is crucial for deepening our understanding of the mechanisms underlying cloud-climate interactions and enhancing the accuracy and reliability of weather forecasts [4] in the context of intensifying global climate change.

Traditional cloud identification methods primarily rely on ground-based observation station data and satellite remote sensing technology, but these methods have significant limitations in terms of spatial resolution and temporal continuity. With technological advancements, developing efficient ground-based automated cloud identification technology has become a critical need. In recent years, the application of devices such as the Whole Sky Imager (WSI), Infrared Cloud Imager (ICI), and Whole Sky Infrared Cloud Measurement System (WSIRCMS) has enabled the automated quantitative identification of ground-based cloud parameters [5]. Among these, WSI and its derivative, the Automatic Cloud Observation System (SONA), provide high-resolution hemispherical sky images in both space and time, enabling the real-time generation of objective and precise cloud cover percentage data, demonstrating significant potential in enhancing weather forecast accuracy [6,7]. Current cloud identification technologies primarily encompass three categories of methods: threshold-based segmentation, traditional machine learning, and deep learning methods [8]. Among these, threshold-based methods exhibit a significant decline in recognition accuracy under complex meteorological conditions such as thin cloud cover, multi-layer cloud systems, or solar flare interference [9]. With the enhancement of computational capabilities, traditional machine learning methods have gradually been applied to the field of cloud recognition [10]. These methods construct classification models based on manually defined texture, shape, and spectral features, enabling the processing of more complex scenarios. However, they are constrained by high dependence on feature engineering and low efficiency in large-scale data processing. In recent years, deep learning methods have made significant progress in cloud recognition due to their automatic feature extraction and efficient big data processing capabilities [11]. In particular, convolutional neural networks (CNNs) can directly learn multi-level abstract features from raw images, enabling high-precision cloud detection and classification [12].

In 2015, CNNs were first applied to small-scale cloud classification tasks, demonstrating that deep convolutional visual features significantly outperformed the state-of-the-art methods at that time in most scenarios [13]. Subsequently, the DeepCloud model was proposed, which was specifically designed for ground-based cloud image classification [14]. A large-scale deep CNN architecture was later introduced and achieved high-precision natural image classification on the ImageNet benchmark dataset [15]. The effectiveness of deep convolutional activation features (DCAFs) for ground-based cloud classification was further validated in [16]. A Deep Multi-modal Fusion (DMF) method was developed to significantly improve cloud classification accuracy by weighting and integrating CNN visual features with multi-modal data [17]. In addition, the CloudNet model was designed specifically for meteorological cloud classification and demonstrated superior performance [18]. Inspired by successful practices in large-scale image classification, an end-to-end trained deep CNN architecture was proposed, providing a new paradigm for addressing small-scale complex visible-light cloud classification tasks [19]. A CNN feature selection fusion system was also constructed, showing outstanding cloud image feature discrimination capability [20]. More recently, a hybrid framework that fuses lightweight CNNs with Transformer networks (CD-CTFM) was proposed, effectively enhancing cloud recognition accuracy [21].

Hainan Island has a typical tropical monsoon climate, maintaining high temperatures and humidity year-round, with significant atmospheric water vapor flux [22]. Under these conditions, low-contrast weather phenomena such as thin clouds and haze frequently occur, leading to blurred cloud map boundaries and reduced cloud type distinguishability. Frequent tropical cyclone activity induces rapid cloud migration and multi-layer superposition phenomena. During storm periods, mirror reflections and Mie scattering caused by strong sunlight further degrade image quality, making it difficult to effectively characterize cloud macro-structural features. Additionally, all-sky imaging instruments are easily affected by strong reflections from water bodies or wet ground surfaces, further complicating cloud identification. Furthermore, the lack of region-specific cloud image datasets severely limits the model’s generalization ability and application scope. Traditional methods that rely on static rule-based modeling have inherent limitations when dealing with the rapid dynamic evolution and irregular structural features of cloud formations in the Hainan region. Deep learning frameworks, however, can significantly improve classification accuracy by learning deep nonlinear feature representations through end-to-end learning. Their application in all-sky cloud image recognition in the Hainan region can significantly enhance the region’s cloud map recognition capabilities.

This paper addresses the intelligent recognition needs of all-sky cloud maps in the Hainan tropical climate zone by introducing a cloud classification method based on CNNs. Considering the region’s high humidity and heat environment, as well as the dominant distribution of cumulus clouds, the study systematically collected, annotated, and validated Hainan-specific all-sky cloud images to construct a cloud map dataset with regional adaptability. Based on this, a CNN architecture adapted to small-sample training paradigms was designed and optimized, effectively improving the classification accuracy of tropical island cloud maps and providing a reliable technical implementation path for meteorological observations.

2. Data and Methods

2.1. Equipment Overview

The dataset used in this paper consists of sky images captured by the all-sky imager SONA 502U (Sieltec Canarias, Las Palmas de Gran Canaria, Spain) between 3 and 30 November 2023, with a temporal resolution of 5 min (Figure 1). This instrument is a high-performance device specifically designed for sky observation by Sieltec Canarias. It is equipped with an 180° fisheye lens and a high-speed CMOS global shutter sensor, with a sampling frequency as fast as 5 s. The device system outputs include image capture, cloud cover percentage, solar obscuration status, and other parameters [23].

2.2. Data Preprocessing

To ensure the training efficiency, convergence speed, generalization ability, and final performance of the CNN model, when constructing the sample dataset for the cloud type recognition model, the image dataset was first rigorously screened to exclude blurry or obstructed images, such as all blurry images and images with rain droplets interfering with the lens (Figure 2). Additionally, considering the time intervals between images, we avoided selecting images with overly short time intervals and high similarity in cloud features. Ultimately, 256 images were carefully selected from the image dataset to form the foundation of this study’s dataset.

Based on the constructed dataset of 256 image samples, meteorological experts classified the images into four types, clear sky, cirrus clouds, cumulus clouds, and stratus clouds (Figure 3), with each category containing 64 images. Each image was treated as a feature vector, and its corresponding cloud type was used as a label, with clear sky, cirrus clouds, cumulus clouds, and stratus clouds labeled as 0, 1, 2, and 3, respectively.

In this study, random sampling was used to divide the 256 images into a training set and a test set in a 5:1 ratio. The training set is used for model training, while the test set is used to evaluate model performance. This division ratio aims to ensure that the model has access to sufficient data during training while effectively assessing its generalization capabilities during the testing phase.

2.3. Methods

This paper uses CNN to identify, classify, and assess cloud cover in cloud images observed by an all-sky imager. The research methods mainly include image preprocessing, cloud classification, and cloud cover detection.

2.3.1. Image Preprocessing

Image preprocessing enhances image quality, laying a solid foundation for subsequent image recognition and analysis tasks. To improve image quality, this paper employs three image preprocessing techniques (Figure 4) to effectively remove image noise, significantly enhance contrast, and sharpen edges. Gaussian filtering reduces image noise levels by applying Gaussian weighting to pixel values while preserving low-frequency image information [24]; Contrast-Limited Adaptive Histogram Equalization (CLAHE) technology intelligently adjusts the histogram distribution to enhance local contrast without causing excessive saturation, thereby highlighting image details [25]; Laplacian filtering technology highlights high-frequency information in images, enhancing the clarity of image contours [26].

2.3.2. Cloud Classification

As a deep learning model, CNN has been widely applied in fields such as image recognition, video analysis, and natural language processing [27]. Therefore, this paper adopts CNN for cloud type identification, with its model diagram shown in Figure 5. A CNN consists of convolutional layers, pooling layers, fully connected layers, activation functions, loss functions, and optimizers.

The convolutional layer is the core component of a CNN, extracting local features from the input image through convolution operations. Pooling layers typically follow convolutional layers, aiming to reduce the spatial dimensions of feature maps, decrease the number of parameters and computational load in subsequent layers, while enhancing the model’s generalization capability. After multiple convolutional and pooling layers, CNNs typically include one or more fully connected layers, which integrate features extracted from previous layers to provide decision-making basis for final classification. Activation functions introduce nonlinear factors into CNNs, applied to the outputs of convolutional and fully connected layers. The loss function measures the difference between the model’s predictions and the true labels, while the optimizer adjusts the network weights to minimize the loss function.

With the rapid development of deep learning, CNNs have become one of the most widely adopted models for cloud classification and other visual recognition tasks due to their strong feature learning capability and robustness [11,12,18].

Unlike previous studies that primarily focus on mid-latitude or continental regions, this study emphasizes all-sky cloud recognition under tropical island climate conditions in Hainan, China, where high humidity, strong solar radiation, and frequent multilayer clouds pose additional challenges. The originality of this work lies in the construction of a region-specific dataset and the targeted optimization of the CNN architecture for small-sample tropical cloud classification.

The proposed CNN architecture consists of three convolutional blocks followed by a fully connected classification head. Each convolutional block includes a 3 × 3 convolutional layer with ReLU activation, followed by a Squeeze-and-Excitation (SE) attention module and a 2 × 2 max-pooling layer.

The numbers of convolutional filters are 32, 64, and 128 for the three blocks, respectively. To reduce overfitting under the small-sample setting, dropout and L2 regularization were applied in the fully connected layer. Data augmentation operations, including random horizontal flipping, rotation, and zooming, were applied online during training. For comparison, the baseline CNN adopts the same network architecture and training strategy, with the SE attention modules removed.

2.3.3. Cloud Cover Detection Method

Cloud cover segmentation employs the K-Means clustering algorithm and Normalized Red-Blue Ratio (NRBR) technology for image segmentation. The K-Means clustering algorithm divides pixels into K clusters, iteratively updates cluster center positions, and performs image segmentation [28]. The NRBR method calculates the ratio based on the red and blue channel ratios, sets a threshold, and segments pixels accordingly [29]. Finally, the cloud cover estimate is obtained by calculating the proportion of pixels in the cloud-covered area.

2.3.4. Cloud Classification Evaluation Metrics

To evaluate the effectiveness of the CNN model in cloud classification, this paper uses precision, recall, and ${\mathrm{F}}_1$ score as evaluation metrics. Precision reflects the accuracy of the model’s predictions, with the formula:

$$Precision={\displaystyle \frac{T_P}{T_P+F_P}},$$

(1)

where $T_P$ (True Positive) refers to the number of samples correctly predicted by the model. $F_P$ (False Positive) refers to the number of samples incorrectly judged by the model. The recall rate measures the percentage of cloud types correctly predicted by the model, and is calculated using the following formula:

$$Recall={\displaystyle \frac{T_P}{T_P+F_N}},$$

(2)

where $F_N$ (False Negative) refers to the number of samples incorrectly predicted by the model. The ${\mathrm{F}}_1$ score is a comprehensive indicator that takes into account both precision and recall, aiming to provide an evaluation value to measure the overall performance of the model. Its calculation formula is:

$$F_1=2\times {\displaystyle \frac{Precision\times Recall}{Precision+Recall}},$$

(3)

The higher the ${\mathrm{F}}_1$ score, the stronger the model’s ability to identify that cloud type. By calculating the evaluation index, the model’s cloud classification and identification performance can be comprehensively assessed.

3. Results

3.1. Cloud Type Identification

To validate the model’s performance, this paper optimizes the model for specific image classification tasks based on a deep learning method using a CNN architecture. Specifically, this includes:

(1)

Freezing the initial two convolutional layers during training to retain general image features, while fine-tuning subsequent convolutional layers to adapt to specific task requirements [30].

(2)

Replacing traditional fully connected layers with global average pooling (GAP) layers [31] to reduce the number of model parameters and mitigate overfitting risks.

(3)

The Adam optimizer is used during model training, and weight decay is introduced to further suppress overfitting [32].

(4)

To enhance the model’s generalization ability, data augmentation strategies are incorporated during training, including random horizontal flipping [33], Gaussian pyramid down-sampling [34], and CLAHE operations [35]. Based on the trained model, the results of four types of cloud recognition obtained by inputting test set samples are shown in Figure 6.

It should be noted that the solar position has a significant influence on cloud segmentation results. When the Sun is located near the field of view center, strong glare and circumsolar brightness may cause misclassification of cloud-free regions, particularly for threshold-based methods such as NRBR and the built-in SONA 502U algorithm, as shown in Figure 7 and Figure 8.

Based on the test set, the trained model was validated, and various validation metrics were calculated as shown in

Table 1. Cloud Recognition Metrics Results.

Cloud Type	Precision	Recall	${\mathbf{F}}_1$ Score
Sunny	1.00	1.00	1.00
Stratus	1.00	1.00	1.00
Cumulus	0.91	0.91	0.91
Cirrus	0.91	0.91	0.91
Average	0.95	0.95	0.95

. The model achieved 100% accuracy, recall, and ${\mathrm{F}}_1$ score in sunny day and stratus cloud classification, mainly due to the significant differences in the characteristics of the two types of clouds: sunny day images have a uniform blue sky background without texture interference, while stratus clouds typically present a continuous low-altitude grayish-white layered structure with smooth edges. In contrast, the classification performance for cumulus and cirrus clouds is slightly lower, which may be attributed to the following reasons:

(1)

Morphological similarity: The blocky contours of cumulus clouds and the filamentous structures of cirrus clouds are easily confused in low-resolution images;

(2)

Lighting interference: Strong sunlight in the Hainan region enhances the reflection at the edges of cirrus clouds, creating local similarities with the fluffy features of cumulus clouds;

(3)

Data limitations: Under small-sample training, the model lacks sufficient learning of fine-grained features.

In summary, the model achieves an average accuracy rate of 95% in cloud type identification, indicating that CNN demonstrates good performance in cloud image recognition.

To further improve the model’s classification performance for cumulus and cirrus clouds, this paper optimizes the network structure by introducing the channel attention mechanism (SEBlock) and multi-scale data augmentation strategies based on the original model.

SEBlock primarily enhances the model’s performance by compressing and exciting input features. Specifically, the SE attention mechanism consists of two steps: Squeeze and Excitation [28]. In the Squeeze step, the input feature map is compressed into a vector via global average pooling, then mapped to a smaller vector through a fully connected layer. In the Excitation step, a sigmoid function is used to compress each element of this vector into the range of 0 to 1, and it is multiplied by the original input feature map to obtain the weighted feature map. The SE attention module is a channel attention module, and the SE module can enhance the channel features of the input feature map without changing its size.

The improved model was retrained on the original dataset, and the validation metrics are shown in

Table 2. Cloud recognition metric results after introducing the SE attention mechanism.

Cloud Type	Precision	Recall	${\mathbf{F}}_1$ Score
Sunny	0.91	0.91	0.91
Stratus	1.00	1.00	1.00
Cumulus	1.00	1.00	1.00
Cirrus	0.91	0.91	0.91
Average	0.95	0.95	0.95

.

An increase in classification accuracy for cumulus clouds is observed, while a slight decrease in clear-sky classification performance is noted. However, given the limited size of the dataset and the small number of samples available for testing each cloud category, these performance variations should be interpreted with caution. Under small-sample conditions, fluctuations in precision, recall, and ${\mathrm{F}}_1$ score may not necessarily reflect statistically significant differences.

Rather than indicating a uniform performance improvement, the introduction of the SEBlock appears to influence the model’s feature emphasis and misclassification tendencies. In particular, the attention mechanism may enhance sensitivity to the aggregated and block-like texture features characteristic of cumulus clouds, while simultaneously increasing sensitivity to subtle texture variations in otherwise homogeneous clear-sky scenes. As a result, some clear-sky samples may be more easily confused with thin-cloud cases.

From a methodological perspective, confusion matrices could provide a more detailed and objective illustration of how misclassification patterns change before and after introducing the attention mechanism. However, given the limited dataset size, the present analysis focuses on overall performance trends rather than statistically rigorous class-wise error analysis.

Overall, these results suggest that channel attention mechanisms such as the SEBlock have the potential to affect cloud feature representation in meaningful ways, particularly for cloud types with complex morphological structures. At the same time, the observed trade-offs highlight the importance of cautious interpretation and the need for larger and more diverse datasets to reliably assess the statistical significance and generalizability of attention-based improvements.

3.2. Total Cloud Cover Detection

In terms of total cloud cover detection, this paper adopts a visual consistency evaluation method, which compares the output of the cloud cover detection algorithm with the visual features of the original image to determine the accuracy of the algorithm. Specifically, we evaluate the performance of different methods by observing the consistency between the cloud regions segmented by the algorithm and the actual cloud distribution in the original image (provided by the device system).

This paper comprehensively evaluates the cloud segmentation performance of K-Means clustering, NRBR (Normalized Red-Blue Ratio), and the built-in software of the SONA 502U (manufacturer-provided embedded software; version not specified in the official user manual). Two representative cases—4 November 2023, at 16:55, and 5 November 2023, at 15:15—were selected to capture typical variations in solar elevation and cloud coverage, enabling a qualitative comparison of segmentation performance under different illumination conditions (Figure 7 and Figure 8), and thereby revealing the advantages and limitations of each method in cloud cover estimation.

For the image captured at 16:55 on 4 November 2023, the K-Means method calculated a total cloud cover of 17.62%. The segmentation results better distinguished blue sky areas from clouds compared to the apparent morphology in the original image, thereby reducing misidentification of clear sky regions as clouds. However, it still underestimated the actual cloud cover by failing to adequately account for the extent of thin or scattered cloud regions. The NRBR method calculated a total cloud cover of 9.56%. The segmented cloud regions closely match the original image’s morphology, clearly distinguishing clouds from the blue sky, and the total cloud cover percentage is closer to the actual value. The total cloud cover calculated by the built-in software of the SONA 502U device is 29.18%. This method exhibits significant misclassification, incorrectly identifying buildings and some non-cloud regions as clouds, resulting in an overly high total cloud cover.

In the example from 5 November 2023, at 15:15, the total cloud cover calculated using K-Means was 49.23%. Although the identified cloud areas covered a large area, some edge areas and blue areas in the sky were misidentified as clouds, resulting in an overestimation of the total cloud cover. The total cloud cover calculated using the NRBR method was 39.89%. The shape of the segmented cloud regions was more consistent with the original image, particularly in terms of the accuracy of cloud boundary details. Compared to the K-Means method, the NRBR method significantly reduces misidentification of non-cloud areas. The total cloud cover identified by the built-in software of the SONA 502U is 62.92%, and there are also cases where buildings are misidentified as clouds, resulting in an overestimated total cloud cover.

Based on the total cloud cover detection results from 4 and 5 November 2023, the NRBR method demonstrates significant advantages in terms of accuracy, robustness, and detail handling. This method constructs segmentation thresholds based on the spectral characteristics of the red and blue channels, enabling precise differentiation between cloud and blue sky regions while effectively capturing cloud morphological details (such as thin cloud edges and fragmented cloud layers), thereby reducing misclassification of non-cloud regions [36]. The K-Means method [28], however, relies on color clustering and lacks spectral feature adaptability, resulting in unstable performance in complex scenarios. For example, on November 4, it underestimated the cloud coverage area, leading to a total cloud cover of only 17.62%, while on November 5, it misjudged sky blue areas as clouds, causing the result to be overestimated to 49.23%. The limitations of the built-in software of the SONA 502U device are even more pronounced, as its algorithm fails to effectively exclude ground interference such as buildings during segmentation (resulting in total cloud cover as high as 29.18% and 62.92% on the two days), causing the total cloud cover to significantly deviate from the actual value. This indicates that the NRBR method, which enhances the contrast between clouds and the background through spectral ratios, is more practical in complex meteorological observation scenarios; the SONA 502U requires further optimization of its non-cloud target recognition logic to reduce the impact of environmental interference on the results.

In summary, the NRBR method performs best in total cloud cover detection, using the ratio of red and blue channels for image segmentation to effectively distinguish between cloud and non-cloud areas, thereby providing a more accurate percentage of total cloud cover. In contrast, the built-in software of the SONA 502U has certain limitations in total cloud cover identification, often misidentifying non-cloud areas such as buildings as clouds, leading to an overestimation of total cloud cover.

4. Conclusions

This paper investigates the application of convolutional neural networks (CNNs) for cloud image classification and cloud cover assessment using observations from the Hainan All-Sky Imager SONA 502U. The main findings can be summarized as follows.

(1)

In terms of cloud identification, the feature-optimized CNN architecture achieved an average classification accuracy of 95%, demonstrating the feasibility and potential of deep learning approaches for all-sky cloud classification. Nevertheless, relatively lower performance for cumulus and cirrus clouds highlights the persistent challenges posed by morphological similarity and illumination interference in fine-grained cloud recognition. After introducing the SEBlock channel attention mechanism, an improvement trend was observed for cumulus cloud classification, suggesting that enhanced channel-wise feature weighting may help emphasize aggregated texture characteristics. However, this improvement was accompanied by a decrease in clear-sky classification performance, indicating a potential trade-off introduced by the attention mechanism under small-sample conditions. Moreover, the difficulty in accurately identifying filamentous cirrus structures under strong illumination remains unresolved, underscoring the limitations of the current approach in modeling lighting invariance. These results should be interpreted with caution due to the limited dataset size.

(2)

In terms of cloud cover detection, visual consistency evaluation methods were employed to compare the performance of different segmentation algorithms. The results indicate that the cloud segmentation method based on the normalized red-blue ratio (NRBR) generally performs better than K-Means clustering and the built-in algorithm of the SONA 502U device. By constructing segmentation thresholds using spectral ratio features derived from the red and blue channels, the NRBR method effectively exploits reflectance differences between clouds and atmospheric backgrounds in the visible spectrum, leading to more consistent cloud–non-cloud separation. Overall, the NRBR-based segmentation shows advantages in terms of morphological coherence and quantitative agreement, although its performance may still be influenced by illumination conditions.

In summary, this study suggests that deep learning-based cloud recognition and classification methods offer promising potential for improving the automation and consistency of all-sky cloud analysis. While the proposed approaches demonstrate encouraging performance, their limitations under small-sample and complex lighting conditions highlight the need for further research. Future work will focus on developing more robust and background-adaptive feature learning strategies, as well as integrating multi-source meteorological data, to enhance the reliability and applicability of automated cloud monitoring for meteorological observation and related applications.