Convolutional Neural Network for High-Resolution Cloud Motion Prediction from Hemispheric Sky Images

: A novel high-resolution method for forecasting cloud motion from all-sky images using deep learning is presented. A convolutional neural network (CNN) was created and trained with more than two years of all-sky images, recorded by a hemispheric sky imager (HSI) at the Institute of Meteorology and Climatology (IMUK) of the Leibniz Universität Hannover, Hannover, Germany. Using the haze indexpostprocessing algorithm, cloud characteristics were found, and the deformation vector of each cloud was performed and used as ground truth. The CNN training process was built to predict cloud motion up to 10 min ahead, in a sequence of HSI images, tracking clouds frame by frame. The ﬁrst two simulated minutes show a strong similarity between simulated and measured cloud motion, which allows photovoltaic (PV) companies to make accurate horizon time predictions and better marketing decisions for primary and secondary control reserves. This cloud motion algorithm principally targets global irradiance predictions as an application for electrical engineering and in PV output predictions. Comparisons between the results of the predicted region of interest of a cloud by the proposed method and real cloud position show a mean Sørensen–Dice similarity coefﬁcient ( SD ) of 94 ± 2.6% (mean ± standard deviation) for the ﬁrst minute, outperforming the persistence model (89 ± 3.8%). As the forecast time window increased the index decreased to 44.4 ± 12.3% for the CNN and 37.8 ± 16.4% for the persistence model for 10 min ahead forecast. In addition, up to 10 min global horizontal irradiance was also derived using a feed-forward artiﬁcial neural network technique for each CNN forecasted image. Therefore, the new algorithm presented here increases the SD approximately 15% compared to the reference persistence model.


Introduction
Short-time cloud motion prediction has a huge impact on the future behavior of the power generation output of solar photovoltaic (PV) power plants [1]. Clouds are a major modulator of the global horizontal irradiance (GHI) and a source of severe fluctuation when, for example, passing in front of the sun. Clouds can even increase the solar radiation at the surface by reflection and/or forward scattering [2][3][4]. To compensate for these ramp events, very short-term forecasting/forecasts can help power plant operators to accurately manage PV power plants. The analyses of clouds play an important role in both scientific and business enterprises, where these severe fluctuations in the energy output are incompatible with the established safety standards for the electricity distribution systems [5].
In this context, the introduction of hemispheric sky imager (HSI) systems as efficient ground surface equipment for cloud data assessment have already been proven by various authors [6]. However, even with good high-resolution cloud detections, cloud movement forecast is still a topic of research due to its high degree of complexity [7][8][9]. Figure 1 shows how quickly cloud changes can occur within three minutes. Looking at cloud detection methods, we can mention threshold-based algorithm [10] and machine learning methods [11][12][13][14][15]. Threshold-based algorithms normally use a red/blue ratio of the three RGB (red, blue, and green) channels from the image pixels for cloud classification [16][17][18][19]. Cloud pixels are identified as high R and B values, while sky pixels have low R and high B values. However, this method has some weaknesses, primarily distinguishing or detecting clouds near the horizon and close to the sun [20]. In addition, different pattern recognition algorithms have been developed. Ren and Malek [21] proposed a cloud segmentation algorithm utilizing superpixel. This algorithm divides the image into blocks (or clusters) and the division is based mainly in the continuity of cloud contours, the texture and brightness of each pixel. A hybrid framework has been proposed to forecast hourly global solar radiation [22]. This approach includes two different methods: support vector machine and machine learning techniques. The results showed that it is possible to predict next-day hourly values of solar radiation by reducing the root mean absolute error ( ) by 15.2% compared to the reference persistence model.
Machine learning methods have been used successfully for cloud detection [23] and cloud coverage estimations [24]. Crisosto [25] developed a method using HSI images to predict cloud concentrations one minute in advance using artificial neural networks (ANNs). The results showed a 30% reduction of errors when compared to the persistence model under diverse cloud conditions. In addition, a similar method has been used as an important step for predicting GHI one hour in advance with one-minute intervals [26]. Other advanced and sophisticated techniques, like convolutional neural networks (CNNs), have been developed and applied in recent years to forecast solar irradiance [27] by offering significant advantages for large image datasets [28], evaluating the non-linearity and other more complex relationships [29].
The main objective of this work is to propose a preliminary pre-processing method required to target solar irradiance predictions that allows companies to make more accurate horizon time predictions. The CNN algorithm can be important for very short-term GHI forecasts, and subsequently better marketing decisions for primary and secondary control reserve (cloud position and the GHI up to 5 min in advance). In this paper, deformation cloud vectors 10 min ahead were determined under different cloud and allweather conditions. We applied an ANN technique to estimate the respective GHI of the forecasted cloud clusters, for methodology validation. Section 2 briefly describes the data acquisition methods. The methodologies of this study are described in Section 3. The results are given in Section 4. The conclusions and future work are discussed in Section 5. Looking at cloud detection methods, we can mention threshold-based algorithm [10] and machine learning methods [11][12][13][14][15]. Threshold-based algorithms normally use a red/blue ratio of the three RGB (red, blue, and green) channels from the image pixels for cloud classification [16][17][18][19]. Cloud pixels are identified as high R and B values, while sky pixels have low R and high B values. However, this method has some weaknesses, primarily distinguishing or detecting clouds near the horizon and close to the sun [20]. In addition, different pattern recognition algorithms have been developed. Ren and Malek [21] proposed a cloud segmentation algorithm utilizing superpixel. This algorithm divides the image into blocks (or clusters) and the division is based mainly in the continuity of cloud contours, the texture and brightness of each pixel. A hybrid framework has been proposed to forecast hourly global solar radiation [22]. This approach includes two different methods: support vector machine and machine learning techniques. The results showed that it is possible to predict next-day hourly values of solar radiation by reducing the root mean absolute error (rMAE) by 15.2% compared to the reference persistence model.
Machine learning methods have been used successfully for cloud detection [23] and cloud coverage estimations [24]. Crisosto [25] developed a method using HSI images to predict cloud concentrations one minute in advance using artificial neural networks (ANNs). The results showed a 30% reduction of errors when compared to the persistence model under diverse cloud conditions. In addition, a similar method has been used as an important step for predicting GHI one hour in advance with one-minute intervals [26]. Other advanced and sophisticated techniques, like convolutional neural networks (CNNs), have been developed and applied in recent years to forecast solar irradiance [27] by offering significant advantages for large image datasets [28], evaluating the non-linearity and other more complex relationships [29].
The main objective of this work is to propose a preliminary pre-processing method required to target solar irradiance predictions that allows companies to make more accurate horizon time predictions. The CNN algorithm can be important for very short-term GHI forecasts, and subsequently better marketing decisions for primary and secondary control reserve (cloud position and the GHI up to 5 min in advance). In this paper, deformation cloud vectors 10 min ahead were determined under different cloud and allweather conditions. We applied an ANN technique to estimate the respective GHI of the forecasted cloud clusters, for methodology validation. Section 2 briefly describes the data acquisition methods. The methodologies of this study are described in Section 3. The results are given in Section 4. The conclusions and future work are discussed in Section 5.

Data
The HSI equipment used was a digital compact charge-coupled device camera and a fish-eye objective with a field of view of 183 • inside a weatherproof box, which provided hemispherical images of the entire sky [30]. Exposure time was 1000/s, and there was an acquisition time of 1 min between each image. In total, 150,000 pictures were produced between 2014 and 2016. From these 150,000 manually segmented images, 5000 were selected for testing (i.e., these pictures were independent of the training data). The system is installed on the roof of the Institute of Meteorology and Climatology (IMUK) of the Leibniz University Hannover in Germany (52.4 • N, 9.7 • E). Completely overcast images were not used in this analysis, since GHI values under 100 W/m 2 are usually not relevant for the production of solar energy and we were more interested in larger GHI ramp effects. In addition, the GHI data was obtained using a CMP11 pyranometer (Kipp & Zonen, Delft, The Netherlands) [31].

Cloud Identification
The method used to identify and separate clouds and sky pixels is an improved sky index image-processing algorithm [32]. The haze index consists of identifying cloud pixels combining the red, blue, and green channels, as detailed by Schrempf [33], and serves as an improvement for hazed areas. Every pixel is then classified as cloudy or clear sky based on a threshold (see Figure 3). Equation (1) presents the haze index, which is applied only to hazed areas, based on thresholds of the sky index.

Semantic Segmentation (Acquiring Labelled Data)
Deep learning and specifically CNNs have drastically improved the way in which intelligent algorithms learn. With convolutional layers, pooling layers, and fully connected layers, CNNs allow computational models to represent data with multiple levels of abstraction [34]. With the automatic cloud-sky separation derived by the haze index algorithm, the automatic cloud segmentation is realized. Therefore, cloud clusters are labelled as ground truth for the automatic segmentation, and thus, for further cloud motion forecasts. Sky clusters were not taken into consideration. Figure 2 shows the process of acquiring the ground truth for the input parameters of the CNN. The first column shows the original image. The results of the haze index algorithm can be seen in the second column. The third column shows the two classes: cloud clusters in white and sky clusters in gray. We can see how the CNN learns to recognize different regions of interest (ROIs) for further simulations. The training process consists of learning how clouds can change frame by frame consecutively.

An Artificial Neural Network Used for Validating Our Model
To validate the results of our algorithm, we applied an extra artificial neural network (ANN), as explained by Crisosto et al. [26]. This ANN only needs an HSI image to predict GHI one hour ahead in one-minute resolution. This input image is the output of our CNN method. Therefore, the new predicted ROIs of our algorithm were fed into the ANN as input parameters to derive their correspondent GHI values 10 min ahead.

Statistical Metrics
The Sørensen-Dice (SD) similarity coefficient [37] and overlap ( ) [38] were used for the method evaluation. SD is defined as the division between twice the number of

An Artificial Neural Network Used for Validating Our Model
To validate the results of our algorithm, we applied an extra artificial neural network (ANN), as explained by Crisosto et al. [26]. This ANN only needs an HSI image to predict GHI one hour ahead in one-minute resolution. This input image is the output of our CNN method. Therefore, the new predicted ROIs of our algorithm were fed into the ANN as input parameters to derive their correspondent GHI values 10 min ahead.

Statistical Metrics
The Sørensen-Dice (SD) similarity coefficient [37] and overlap (VO) [38] were used for the method evaluation. SD is defined as the division between twice the number of elements common to both sets and the sum of the number of elements in each set (Equation (2)). VO is defined as a quotient of the intersection of both X and Y segmentations (Equation (3)) Energies 2021, 14, 753 where |X| and |Y| is the cardinality of two sets. The mathematical definition of the root mean square error (RMSE) and the coefficient of determination (R 2 ) are expressed as follows: where y i is the forecast value, x i is the measured value, and N is the total number of samples. Additionally, x = ∑ N i=1 x i and y = ∑ N i=1 y i . The new algorithm was compared with the scaled persistence model [39] defined as the ROI configuration vector (ROI_CV) where the next minute would be identical to the current minute (ROI_CV t+1 = ROI_CV t ). This model is the reference model for short-term solar forecasting [40]. For the irradiance evaluation, only the movement of the sun was taken into consideration.

Results
For a better visual representation of the results, the following example cases were selected to show the effectiveness and efficiency of the algorithm: 15 April 2014 at 15:54, 11 July 2014 at 13:09, and 1 June 2014 at 17:42. After that, 5000 cloud images with different cloud positions were simulated and the results are presented. Figure 3 shows the one-minute ahead simulation for the three days, including the observed (target) ROI to be simulated (column 1), the simulated ROI by our model (column 2), and the simulated ROI using the persistence model (column 3). In all cases, we can see that our CNN model performs better than the persistence for the first minute. The SD values were 92.4 ± 2.9% (mean ± standard deviation), 92.7 ± 2.6%, and 88 ± 3.4%, respective to each case, while the SD for the persistence was 85.8 ± 4.9%, 85.8 ± 5.6%, and 84 ± 7.1%, respectively.

Analysis of the Example Cases
The statistical comparison with forecasts up to 10 min ahead can be seen in Table 1. As expected, the performance of the algorithm decreases substantially as the simulated time progresses. In addition, our model outperforms the persistence model for the full 10-min period. For larger timescale forecasts (for example, 5 and 10 min), these results were already expected; however, for very short forecasts (for example, 1 and 2 min), our CNN shows improvements in forecasting cloud movement.

Application of the Presented CNN Algorithm
To validate our algorithm, we applied an ANN as described in Section 3.4. For each predicted image from our CNN, the GHI value was predicted at the same time. Figure 4 shows a comparison between the target images and simulated images one-minute ahead generated as output by the CNN and their correspondent measured and simulated GHI. Figure 5 shows the results for two examples of 10-min ahead simulations utilizing our method as an irradiance simulations tool. Table 3 shows a comparison between 100 observed images with their corresponding simulated values. Figure 6 shows the distribution of the relative deviations as a boxplot for different time horizons.  The statistical comparison with forecasts up to 10 min ahead can be seen in Table 1. As expected, the performance of the algorithm decreases substantially as the simulated time progresses. In addition, our model outperforms the persistence model for the full 10min period. For larger timescale forecasts (for example, 5 and 10 min), these results were already expected; however, for very short forecasts (for example, 1 and 2 min), our CNN shows improvements in forecasting cloud movement.     Table 2 presents the statistical indicators of the cloud ROI changes in the 5000 tested images. Tables 1 and 2 show the quality of the results decreasing in forecasts for longer time scales, with SD values up to 44.4 ± 12.3% and VO of 37.7 ± 15.3% obtained for 10min ahead forecasts. Table 2. Comparison between the statistical indicators of the proposed CNN model and the persistence model: the mean Sørensen-Dice similarity coefficient (SD) and overlap (VO) of the 5000 simulated cases for 1-, 2-, 5-, and 10-min forecasts.

Application of the Presented CNN Algorithm
To validate our algorithm, we applied an ANN as described in Section 3.4. For each predicted image from our CNN, the GHI value was predicted at the same time. Figure 4 shows a comparison between the target images and simulated images one-minute ahead generated as output by the CNN and their correspondent measured and simulated GHI.    Table 3 shows a comparison between 100 observed images with their corresponding simulated values. Figure 6 shows the distribution of the relative deviations as a boxplot for different time horizons.   Table 3 shows a comparison between 100 observed images with their corresponding simulated values. Figure 6 shows the distribution of the relative deviations as a boxplot for different time horizons.

Original images with their measured GHI
(a) (b)   Here, there are narrower interquartile ranges for higher sample sizes; symmetry in 50% of the data decreases as soon as the program receives more information, but the numbers of outliers (+) are lower.

Conclusions
In this study, a novel method to forecast cloud motion up to 10 min ahead was presented. A convolutional neural network (CNN) was trained using hemispherical sky images as inputs, and a statistical approach for forecasting future cloud motion was per-

Conclusions
In this study, a novel method to forecast cloud motion up to 10 min ahead was presented. A convolutional neural network (CNN) was trained using hemispherical sky images as inputs, and a statistical approach for forecasting future cloud motion was performed.
According to the simulation results, the method presented here is capable of predicting cloud changes for the first minute with very high confidence using CNN, with a coefficient of determination (R 2 ) of 0.97 and a Sørensen-Dice similarity coefficient (SD) of 94 ± 2.6%. For the same simulated datasets, the persistence model reached an R 2 of 0.92 and a SD of 89 ± 3.8%. The method was also tested for different forecast time scales, however, unsatisfactory results (an R 2 of 0.40 and a SD of 49 ± 11.8%) were obtained for 10-min simulations by our model, although they were better than the results of the persistence model. In addition, the global horizontal irradiance (GHI) output results predicted by the CNN showed a forecast accuracy for the decreased amount of energy one-minute ahead, achieving a RMSE of 32 (W/m 2 ) and a R 2 of 0.81. The persistence model achieved a RMSE of 45 (W/m 2 ) and a R 2 of 0.76. However, for the GHI prediction for the next 10-min ahead, the RMSE was 148 (W/m 2 ) and the R 2 was 0.42 for our model and the RMSE was 187 (W/m 2 ) and the R 2 was 0.39 for the persistence model.
The research presented here can be used as a first step for PV companies to understand cloud movement and to implement an end-to-end forecasting system (as a pipeline) within a fully automated server with the goal of forecasting global horizontal irradiance minutes ahead. This fully automated pipeline implementation will help to allow PV companies to make accurate horizon time predictions and better marketing decisions for primary and secondary control reserves (i.e., up to 5 min in advance).
Future research is needed to better understand cloud movement through wind speed and wind direction, and also to understand how to improve forecast results for periods longer than 1 min or when the sky is totally covered. Different methodologies and maybe different analyses of data should be considered.
Despite the good results, the existence of other models offers new ways to process big data. For example, long short-term memory networks (LSTMs) appear to be an alternative. Since the architecture of these networks is more complex, LSTMs are suitable for processing long data sequences while avoiding vanishing or exploding gradients, currently problems that CNNs still have. As an outlook for further projects, the utilization of LSTM and hybrid models should be taken into consideration.