Short-Term Solar Radiation Prediction Based on Convolution Neural Network and Fitted Clear-Sky Model

Abstract

This study proposes an advanced short-term Direct Normal Irradiance (DNI) prediction model for Concentrated Solar Power (CSP) systems, integrating a convolutional neural network (CNN) with a fitted clear-sky DNI model. Leveraging all-sky images and historical DNI data, the model precisely identifies cloud motion patterns through dense optical flow analysis and forecasts DNI using a targeted region-of-interest (ROI) approach. When maximum cloud pixel velocity falls below 5 pixels per minute, the clear-sky DNI model or persistence model directly applies; for higher-velocity conditions, the CNN predicts the clear-sky index to dynamically adjust the forecast. Experimental validation across diverse weather conditions demonstrates superior accuracy, achieving significantly lower normalized Mean Absolute Errors (nMAEs) and normalized Root Mean Squared Errors (nRMSEs) for various forecast horizons under cloudy skies compared to recent state-of-the-art deep learning approaches. This work delivers a robust solution for preventing thermal shock in the receiver and improving the CSP operational stability.

1. Introduction

In recent decades, solar energy has emerged as a prominent renewable and environmentally friendly energy source, capturing widespread attention owing to its eco-benefits and cost-effectiveness. As its utilization escalates, particularly in large-scale solar power plants, the importance of precision in solar radiation forecasting has become increasingly evident. Among the various components of solar energy systems, heliostat field systems—comprising numerous reflective mirrors (heliostats) [1] directing sunlight to a central receiver for electricity generation—stand out. Yet, the deployment of Concentrated Solar Power (CSP) technologies, including heliostat systems, encounters numerous scientific and technical hurdles, notably the effective control of CSP operations. Although the short-term Direct Normal Irradiance (DNI) forecasting model proposed in this study is generalizable to other scenarios requiring accurate DNI data, such as conventional photovoltaic (PV) power generation systems, it holds unique and irreplaceable significance for CSP systems due to their distinct operational constraints. Unlike PV systems with relatively high tolerance to irradiance fluctuations, CSP systems are highly vulnerable to sudden DNI variations triggered by cloud cover: such abrupt irradiance ramps induce severe thermal shocks to the central receiver, triggering rapid and extreme temperature swings, accelerating receiver component degradation, causing heat transfer tube blockages, and even leading to unplanned system shutdowns, all of which severely compromise system stability and operational efficiency [2,3,4]. Notably, most modern CSP plants are equipped with molten salt thermal storage systems, which reduce the dependence on high-precision DNI forecasting under stable clear or overcast sky conditions; instead, the accurate prediction of sudden DNI fluctuations caused by dynamic cloud motion has become the paramount operational requirement to avoid critical operational hazards. Accordingly, the development of an advanced short-term DNI forecasting model tailored to these dynamic conditions is essential, aiming to effectively mitigate thermal shock risks and further improve the operational efficiency, stability, and overall market competitiveness of CSP technologies.

Traditional prediction methods for DNI primarily rely on statistical approaches [5,6]. Statistical methods, such as time series analysis and regression analysis, predict DNI by exploring the relationship between historical data and solar irradiance [5,6]. However, these methods struggle to handle the complex non-linear relationships between various meteorological variables and solar irradiance, thereby limiting further improvement in prediction accuracy. Recurrent neural network (RNN) and its variants [7], such as long short-term memory (LSTM) and gated recurrent unit (GRU), have shown the potential to improve the prediction of solar irradiance. Convolutional neural networks (CNNs) and transformers are also utilized to deal with time series to forecast DNI, Global Horizontal Irradiance (GHI), or PV power generation [7,8,9]. However, these methods are weak at detecting the DNI ramps due to the cloud movement.

All-sky imagers are commonly deployed at CSP plants to monitor and collect cloud imagery. Consequently, DNI prediction methods leveraging these images have been developed. Intra-hour DNI forecasting techniques based on cloud tracking [10,11] estimate cloud speed and direction through continuous monitoring, thereby predicting cloud cover duration and subsequent DNI changes. Chu et al. [12] initially manually extracted features from all-sky images before employing an artificial neural network for DNI prediction; however, significant information loss during manual feature extraction resulted in limited prediction accuracy. A more prevalent strategy involves the adoption of CNNs or vision transformers. Feng et al. [13] proposed the SolarNet framework, a CNN-based model for intra-hour solar forecasting, enabling end-to-end feature extraction directly from all-sky imagery. Zhao et al. [14] developed a 3D-CNN approach that fuses multi-temporal image sequences with historical meteorological data. Zhu et al. [15] further enhanced hourly DNI prediction by integrating a Siamese CNN with LSTM to capture both spatial cloud features and temporal dynamics. Nevertheless, the model’s reliance on air mass measurements, a parameter rarely monitored in operational CSP plants, hinders its practical engineering application. Karout et al. [16] addressed the limitation of air mass data dependency by forecasting clear-sky DNI directly from field measurements and further proposed a hybrid DNI prediction framework combining dense optical flow-based cloud motion tracking, KNN-driven cloud region segmentation, and CNN-MLP hybrid regression networks. This framework improved prediction accuracy by refining cloud movement tracking and cloud region identification within all-sky images, eliminating the need for hard-to-obtain air mass parameters. Recently, vision transformers have also been applied to forecast solar irradiance or photovoltaic generation [17,18,19,20].

Despite the advantages of the aforementioned hybrid framework [16], it suffers from a critical drawback in its training strategy: utilizing mixed datasets that include low-dynamics clear-sky and overcast conditions introduces irrelevant noise for cloud motion dynamics modeling, which ultimately restricts prediction accuracy under rapidly changing cloudy skies. To overcome this limitation, our study enhances DNI prediction accuracy by refining data preprocessing and model training strategies. Specifically, we introduce a cloud-speed-based data partitioning approach, training the deep-learning model exclusively on datasets where the maximum cloud pixel velocity exceeds 5 pixel/min—ensuring robust representation of dynamic cloud behaviors. Additionally, we propose an hourly parameter fitting method for the clear-sky DNI model, enabling precise calculation of intra-hour clear-sky DNI values and corresponding DNI clear-sky indices during training, while adopting the same clear-sky DNI calculation method as Karout et al. during testing. Our framework applies cloud motion analysis to differentiate between cloud-obstructed and cloud-free situations. For cloud-free periods, the predicted clear-sky DNI directly serves as the forecast; for cloudy conditions, a CNN model processes both cloud-identified regions and corrected images to predict the clear-sky index, which is then multiplied by the clear-sky DNI to yield the final DNI forecast.

The remainder of this paper is organized as follows: Section 2 describes the data collection process, elucidates the clear-sky DNI forecast methodology, presents the experimental data processing techniques, and details the DNI forecast model architecture. Section 3 evaluates the experimental results. Section 4 analyzes the comprehensive performance of the DNI prediction model. Finally, Section 5 synthesizes the key conclusions.

2. Materials and Methods

2.1. Data Collection

All measured data in the daytime were collected in the Hami Pilot CSP. The plant is located at 42.890° N and 93.575° E, 848 m above sea level in Hami City, Xinjiang, China. The DNI data were taken from the meteorological data in the plant with a 1 min sampling frequency with the sensitivity of 8.85 μV/(W/m²) and the uncertainty of 0.13 μV/(W/m²) corresponding 1.44%.

The all-sky images used were RGB images obtained using an all-sky camera (SONA 202U), and the image resolution was 1158 × 1172 pixels. The all-sky images are obtained with 1 min sampling frequency. Furthermore, the all-sky images were excluded when the solar zenith angle exceeded 80 degrees, to prevent inclusion of hazy skies and potential obstructions [21].

2.2. Clear-Sky DNI Forecast

2.2.1. Clear-Sky DNI Model

The clear-sky DNI model developed by Nou et al. [22] is employed to generate clear-sky DNI predictions. The clear-sky DNI at time t can be expressed as follows:

$${\mathrm{D}\mathrm{N}\mathrm{I}}_{\mathrm{c}\mathrm{s}}\left(t\right)=b{I}_0\exp \left(-0.09m\left(t\right)\left(T_{\mathrm{L}\mathrm{I}}\left(t\right)-1\right)\right),$$

(1)

where ${\mathrm{D}\mathrm{N}\mathrm{I}}_{\mathrm{c}\mathrm{s}}\left(t\right)$ represents the clear-sky DNI with unit W/m², the dimensionless parameter b is a function of the considered site’s altitude, I₀ is the extraterrestrial solar irradiance with unit W/m², the dimensionless parameter m$\left(t\right)$ is the optical air mass, and the dimensionless parameter T_LI$\left(t\right)$ denotes the Linke turbidity coefficient. b is given by the following:

$$b=0.664+0.163·e^{h/8000}b=0.664+0.163·e^{h/8000},$$

(2)

where h is the site’s altitude. I₀ can be expressed as follows [23]:

$$I_0={I\prime }_0·\left[1+0.0334·\cos \left(2\mathsf{\pi }·doy/365\right)\right],$$

(3)

where the solar constant I′₀ is set to 1361.2 Wm⁻² and doy is the day of year, ranging from 1 (1 January) to 365 (31 December). The optical air mass m is given by the following:

$$m\left(t\right)={\left[\cos \left(zen\left(t\right)\right)+0.50572·{\left(96.07995-zen\left(t\right)\right)}^{-1.6364}\right]}^{-1},$$

(4)

where zen is the Sun zenith angle. Thus, only T_LI needs to be determined. This parameter can be fitted based on the historical clear-sky DNI data from t − Δt to t, and the fitted parameter can be used to predict clear-sky DNI values from t to t + Δt.

2.2.2. Clear-Sky Detection

There are various existing clear-sky detection algorithms [24,25,26,27], but some of them [24,25] are considered overly complex for real-time applications. Reno and Hansen [26] proposed a simple and computationally efficient clear-sky detection algorithm that can classify Global Horizontal Irradiance (GHI) measurements into clear-sky and non-clear-sky datasets. Based on the method of Reno and Hansen [26], Bone et al. [27] proposed an even simpler approach that also yields good results. We have made modifications to the method proposed by Bone et al. and have achieved similarly excellent outcomes. The procedures are as follows.

(1)

Selecting the First Clear-Sky DNI Value within a Time Period.

Firstly, shift the timestamp corresponding to the DNI data point at time t backward by $\Delta t$, where $\Delta t$ = 1 min. Calculate the solar zenith angle at time $t-\Delta t$. Combined with the preset initial turbidity value, compute the estimated clear-sky DNI value, ${DNI}_{\mathrm{c}\mathrm{s},\mathrm{g}\mathrm{u}\mathrm{e}\mathrm{s}\mathrm{s}}\left(t-\Delta t\right)$, at this $t-\Delta t$ moment. The initial turbidity value is taken as 8.

Determine the solar zenith angle, $zen\left(t\right)$, at the current time corresponding to the DNI data point. If $zen\left(t\right)>zen\left(t-\Delta t\right)$, where $DNI\left(t\right)\ge {DNI}_{\mathrm{c}\mathrm{s},\mathrm{g}\mathrm{u}\mathrm{e}\mathrm{s}\mathrm{s}}\left(t-\Delta t\right)·{thre}_0$ is met, confirm it as the first clear-sky DNI value of this time period, and record the corresponding zen(t). Here, ${thre}_0$ is a threshold value between 0 and 1. In our tests, ${thre}_0=0.94$ proved effective.

If $zen\left(t\right)\le zen\left(t-\Delta t\right)$, when $DNI\left(t\right)\ge {DNI}_{\mathrm{c}\mathrm{s},\mathrm{g}\mathrm{u}\mathrm{e}\mathrm{s}\mathrm{s}}\left(t-\Delta t\right)·{thre}_1$ is encountered, identify this as the first clear-sky DNI value within the current time period and record the corresponding $zen\left(t\right)$. Consequently, $DNI\left(t\right)$ is selected as ${DNI}_{cs}\left(t\right)$. Here, ${thre}_1$ represents a threshold value ranging from 0 to 1. In our tests, a value of ${thre}_1=0.98$ proved appropriate.

If the DNI data point fails both criteria, proceed to the next data point in the time period and repeat the steps.

(2)

Selecting Subsequent Clear-Sky DNI Values within a Time Period.

Assume the last selected clear-sky DNI and solar zenith angle in the current time period are ${DNI}_{cs}\left(t\prime \right)$ and $zen\left(t\prime \right)$, respectively, and the DNI and solar zenith angle of the current data point are $DNI\left(t\right)$ and $zen\left(t\right)$, respectively. Here, $t\prime$ represents the time of the last point in the selected list.

If $zen\left(t\right)>zen\left(t\prime \right)$, where $DNI\left(t\right)\ge {DNI}_{cs}\left(t\prime \right)·{thre}_0$ is met, add $DNI\left(t\right)$ to the clear-sky DNI value list, and record the corresponding $zen\left(t\right)$.

If $zen\left(t\right)\le zen\left(t\prime \right)$, where $DNI\left(t\right)\ge {DNI}_{cs}\left(t\prime \right)·{thre}_1$ is met, add $DNI\left(t\right)$ to the clear-sky DNI value list, and record the corresponding $zen\left(t\right)$.

If the DNI data point satisfies neither criterion, omit $DNI\left(t\right)$ from the clear-sky DNI value list.

Repeat these steps for the next data point in the time period until all DNI data in the time period have been processed. Finally, obtain a list of clear-sky DNI values and solar zenith angles, i.e.,

$$\left[\left({DNI}_{cs}\left(t_0\right),zen\left(t_0\right)\right),\left({DNI}_{cs}\left(t_1\right),zen\left(t_1\right)\right),\dots ,\left({DNI}_{cs}\left(t_n\right),zen\left(t_n\right)\right),\right]$$

where $t_0,t_1,\dots ,t_n$ represent the first, second, …, and last time points corresponding to the data in the selected clear-sky list, respectively, and $t_0<t_1<\dots <t_n$. ${DNI}_{cs}\left(t_0\right)$ and $zen\left(t_0\right)$, ${DNI}_{cs}\left(t_1\right)$ and $zen\left(t_1\right)$, …, and ${DNI}_{cs}\left(t_n\right)$ and $zen\left(t_n\right)$ represent the clear-sky DNI and solar zenith angle at these respective time points.

Two comprehensive flowcharts (Figures S1 and S2) have been added to the Supplementary Materials, delineating the clear-sky detection algorithm.

2.2.3. Clear-Sky DNI Model Fitting

We employ different clear-sky DNI model fitting strategies during the CNN model training and when using the trained CNN model for clear-sky index prediction.

During the CNN model training, the clear-sky DNI used is obtained by hourly fitting of clear-sky DNI selected through the method described in Section 2.2.2 at each time point. If the number of clear-sky DNI data points within an hour is fewer than 5, the optical air mass parameter defaults to a value of 8.

When using the trained CNN model as described in Section 2.4.3 for DNI clear-sky index prediction, if the current DNI value satisfies the criteria for selecting clear-sky DNI as outlined in Section 2.2.2, the current DNI value is used to calculate the Linke turbidity coefficient TLI. Otherwise, the Linke turbidity coefficient TLI from the previous time step is utilized to compute the clear-sky DNI value for subsequent time steps.

2.3. Data Preprocessing

2.3.1. DNI Clear-Sky Index

We obtain the clear-sky DNI values at each time point in the dataset using the method described in Section 2.2.3, and then divide the DNI values by the clear-sky DNI values to derive the DNI clear-sky index.

2.3.2. Distortion Correction

The all-sky imager is essentially a fisheye camera, thus requiring distortion correction for its images. The methods for distortion correction have been detailed in the literature [28,29] and will not be reiterated here.

2.3.3. Cloud Motion Estimation

Cloud motion estimation serves as the core foundation for capturing DNI ramps induced by cloud movement, with three mainstream technical paradigms: cross-correlation methods (CCM) [11,30,31], particle image velocimetry (PIV) [10,32,33], and dense optical flow algorithms. CCM, as a classical and computationally efficient method, infers global cloud motion by correlating sequential all-sky image pairs, usually preprocessed with a 5 min moving average filter [31] to reduce background noise; Quesada-Ruiz et al. [11] enhanced its applicability by introducing angular-radial grid-based cloud masks, which improves the robustness of bulk motion estimation for irregular cloud shapes. Derived from classical fluid dynamics principles, PIV tracks cloud features as discrete particle-like tracers to derive motion fields, a technique first introduced for cloud monitoring by Mori and Chang [32] and refined by Chu et al. [33]. Marquez and Coimbra [10] enhanced PIV performance for all-sky imagery via k-means clustering, which effectively isolates dominant cloud motion vectors and eliminates interference from trivial small-scale features. In contrast to sparse vector estimation of the former two methods, dense optical flow algorithms [34,35] achieve pixel-level motion field estimation by leveraging spatiotemporal intensity gradients, based on core assumptions of brightness constancy and spatial coherence. This technique has been widely validated for cloud tracking: Du et al. [36] verified its feasibility using Sun et al.’s optical flow model [34], while Farnebäck’s dense optical flow algorithm [35] has become a mainstream choice in DNI forecasting, adopted by West et al. [37] and Karout et al. [16] for high-precision cloud movement identification.

In this study, we employ the Farnebäck algorithm [35] to calculate the velocity magnitude and direction for each pixel, as detailed below:

First, the image is converted to grayscale through linear transformation into the HSV color space, utilizing the brightness dimension V as the grayscale information:

$$V=\mathrm{m}\mathrm{a}\mathrm{x}\left(R,G,B\right),$$

(5)

where R, G, and B represent the luminance values of the red, green, and blue channels in the RGB color space, respectively.

The Farnebäck algorithm [35] treats the grayscale value of an image pixel as a function of two variables, fx,y. A local coordinate system is constructed centered on the pixel of interest, and the function is approximated using a quadratic expansion, expressed as follows:

$$f\left(x,y\right)=f\left(\mathbf{x}\right)={\mathbf{x}}^{\mathrm{T}}\mathbf{A}\mathbf{x}+{\mathbf{b}}^{\mathrm{T}}\mathbf{x}+c,$$

(6)

Here, $\mathbf{x}$ is a two-dimensional column vector, A is a 2 × 2 symmetric matrix, b is a 2 × 1 matrix, $f\left(\mathbf{x}\right)$ is equivalent to $f\left(x,y\right)$ representing the pixel’s grayscale value, and c denotes the constant term of the quadratic expansion. If this pixel moves with a displacement e, the polynomial coefficients change. Assuming A remains constant before and after displacement, the functions before and after the change are, respectively:

$$\begin{gathered} f_1\left(\mathbf{x}\right)={\mathbf{x}}^{\mathrm{T}}\mathbf{A}\mathbf{x}+{{\mathbf{b}}_1}^{\mathrm{T}}\mathbf{x}+c_1, \\ {f}_2\left(\mathbf{x}\right)={\mathbf{x}}^{\mathrm{T}}\mathbf{A}\mathbf{x}+{{\mathbf{b}}_2}^{\mathrm{T}}\mathbf{x}+c_2, \end{gathered}$$

(7)

where ${\mathbf{b}}_1$ and ${\mathbf{b}}_2$ represent the 2 × 1 matrices before and after the change, and $c_1$ and $c_2$ represent the constant terms before and after the change, respectively. This leads to the constraint equation:

$$\mathbf{A}\mathbf{e}=∆\mathbf{b},$$

(8)

where

$$∆\mathbf{b}={\displaystyle \frac{{\mathbf{b}}_2-{\mathbf{b}}_1}{2}}.$$

(9)

Finally, an objective function is established:

$${‖\mathbf{A}\mathbf{e}-∆\mathbf{b}‖}^2.$$

(10)

The displacement vector $\mathbf{e}$ is solved by minimizing this objective function. The velocity vector is then obtained by dividing the displacement $\mathbf{e}$ by the time interval between frames.

2.3.4. Region of Interest

By employing the Farnebäck algorithm [35], we can obtain the pixel velocity of each point on the moving cloud cluster in the image. Due to inherent errors in the algorithm itself, even for points within the same cloud cluster, the calculated pixel velocities exhibit significant variations. Our objective is to determine the overall velocity of the entire cloud cluster, thus necessitating the derivation of the dominant cloud cluster velocity from the image velocities of individual points output by the Farnebäck algorithm. In previous works [10,11,16,33,38], the k-means clustering algorithm [39] and one-class k-means clustering have been utilized to identify the dominant velocity of cloud motion.

In this work, through the analysis of typical distortion-corrected all-sky imaging under cloudy weather conditions, we found that, due to the sensitivity of dense optical flow methods to light [35], cloud-free regions are also detected as having motion. Through manual annotation, it was determined that, in these typical cases, regions with pixel velocities below 5 pixels per minute are generally cloud-free areas. Therefore, we averaged the velocities of regions with pixel velocities greater than or equal to 5 pixels per minute to obtain the dominant cloud motion velocity.

By employing a solar position calculation algorithm [40] combined with the camera imaging model, the position of the sun in the distortion-corrected image can be computed. This allows us to determine which region, under the current dominant cloud motion velocity, may exhibit cloud occlusion of the sun after a certain period. This region is designated as the region of interest (ROI). In our model, the optimal performance is achieved when the ROI is set as a square with a side length of 88 pixels. Details can be found in the Supplementary Materials. This size approximately corresponds to the circumscribed square of the sun in the image. In the work of Karout et al. [16], the optimal ROI area was 24 times the size of the solar region. The difference in these regions primarily arises from the use of different models, with detailed discussions provided below.

When the ROI region overlaps with the solar region, the overlapping pixels in the ROI region are set to 0. Testing revealed that the optimal effect is achieved when the solar region is defined as a square with a side length of 88. Details are provided in the Supplementary Materials.

2.4. DNI Forecast Model

2.4.1. The Primary Framework

In previous works [10,11,12,13,14,15,16,38], various weather conditions were typically handled with a uniform model. Since all-sky images differ significantly under clear, overcast, and cloudy skies, the model struggles to balance these variations, resulting in reduced prediction accuracy. In this study, we adopt distinct models for different conditions based on whether the maximum cloud pixel velocity calculated by the dense optical flow method is less than 5 pixels per minute, $\left|v_{\mathrm{m}\mathrm{a}\mathrm{x}}\right|<5$ (as shown in Figure 1). If $\left|v_{\mathrm{m}\mathrm{a}\mathrm{x}}\right|<5$, a rule-based “still model” is applied; otherwise, a convolutional neural network (CNN) model termed the “motion model” is employed. $\left|v_{\mathrm{m}\mathrm{a}\mathrm{x}}\right|<5$ typically indicates clear-sky or overcast conditions, while $\left|v_{\mathrm{m}\mathrm{a}\mathrm{x}}\right|\ge 5$ corresponds to cloudy conditions.

2.4.2. The Still Model

As shown in Figure 2, the still model for DNI forecasting operates as follows:

If the current DNI at time t, $DNI\left(t\right)$, is less than $DN{I}_{thre}$, the predicted DNI at $t+H$, $DN{I}_{pred}\left(t+H\right)$, is set equal to $DN{I}_t$, where H denotes the forecast horizon. Otherwise, when $DNI\left(t\right)\ge DN{I}_{thre}$ the following steps are completed:

• The Linke turbidity coefficient, $T_{LI,t}$, is calculated from the current DNI using the inverse function, $f^{-1}$, such that $T_{LI}\left(t\right)=f^{-1}(t,DNI\left(t\right))$.
• The predicted DNI, $DN{I}_{pred}\left(t+H\right)$, is computed by applying the function $f$ at $t+H$ using the derived $T_{LI}\left(t\right)$, expressed as $DN{I}_{pred}\left(t+H\right)=f(t+H,T_{LI}\left(t\right))$.

Figure 2. The still model for the DNI forecasting operates as follows: If the current DNI falls below $DN{I}_{thre}$, $DNI\left(t\right)<DN{I}_{thre}$, the predicted DNI, $DN{I}_{pred}\left(t+H\right)$, is set to $DNI\left(t\right)$; otherwise, the Linke turbidity coefficient, $T_{LI}\left(t\right)$, is calculated according to the current DNI, and then the predicted DNI, $DN{I}_{pred}\left(t+H\right)$, is calculated using the calculated coefficient. The function f corresponds to Equation (1), i.e., $f(t,T_{LI}\left(t\right))={\mathrm{D}\mathrm{N}\mathrm{I}}_{\mathrm{c}\mathrm{s}}\left(t\right)=b{I}_0\exp \left(-0.09m\left(t\right)\left(T_{LI}\left(t\right)-1\right)\right)$, and the function $f^{-1}$ is the inverse function of $f$.

Figure 2. The still model for the DNI forecasting operates as follows: If the current DNI falls below $DN{I}_{thre}$, $DNI\left(t\right)<DN{I}_{thre}$, the predicted DNI, $DN{I}_{pred}\left(t+H\right)$, is set to $DNI\left(t\right)$; otherwise, the Linke turbidity coefficient, $T_{LI}\left(t\right)$, is calculated according to the current DNI, and then the predicted DNI, $DN{I}_{pred}\left(t+H\right)$, is calculated using the calculated coefficient. The function f corresponds to Equation (1), i.e., $f(t,T_{LI}\left(t\right))={\mathrm{D}\mathrm{N}\mathrm{I}}_{\mathrm{c}\mathrm{s}}\left(t\right)=b{I}_0\exp \left(-0.09m\left(t\right)\left(T_{LI}\left(t\right)-1\right)\right)$, and the function $f^{-1}$ is the inverse function of $f$.

The function $f$ corresponds to Equation (1), defined as follows: $f(t,T_{LI}\left(t\right))=b\cdot I_0\cdot \exp \left[-0.09\cdot m\left(t\right)\cdot (T_{LI}\left(t\right)-1)\right]$, and $f^{-1}$ denotes the mathematical inverse of $f$. This formulation enables dynamic adjustment of atmospheric clarity effects during high-irradiance conditions.

Through testing, we found that setting the DNI threshold, $DN{I}_{thre}$, to 15 W/m² is optimal. Details can be found in the Supplementary Materials.

2.4.3. The Motion Model

The motion model designed for Direct Normal Irradiance (DNI) forecasting employs a convolutional neural network (CNN) architecture to process sky imagery data and predict the DNI clear-sky index, as shown in Figure 3. This model integrates two complementary visual inputs: a region of interest (ROI) image (3 × 88 × 88) capturing specific sky sections, as described in Section 2.3.4, and a distortion-corrected all-sky image (3 × 1000 × 1000) providing a hemispheric context. The processing pipeline begins with the downsampling of the distortion-corrected all-sky image to precisely match the pixel dimensions of the ROI image (3 × 88 × 88). This alignment enables pixel-level fusion of the two inputs into a unified 6-channel tensor (6 × 88 × 88). The first three channels originate from the ROI image (typically RGB), while the subsequent three channels correspond to the resized all-sky image.

This composite tensor is fed into a sequence of 4 convolutional layers, each performing hierarchical feature extraction through learned filters. Following each convolution, a Rectified Linear Unit (ReLU) activation function introduces non-linearity by outputting the positive component of the input $f\left(x\right)=\mathrm{m}\mathrm{a}\mathrm{x}(0,x)$, thereby enabling the model to capture complex, non-linear relationships between cloud dynamics and solar irradiance. Subsequently, max-pooling operations with 2 × 2 windows spatially downsample the feature maps, reducing dimensionality while retaining dominant features. This stride-based reduction enhances translational invariance and computational efficiency.

After the convolutional blocks, the extracted spatial features are flattened into a 1D vector and propagated through two fully connected (dense) layers. These layers perform high-level feature integration and regression analysis, progressively narrowing the data representation to isolate patterns predictive of solar irradiance modulation. The final layer outputs a scalar value representing the DNI clear-sky index ($k_{cs}$)—a normalized metric quantifying the attenuation of direct sunlight due to cloud cover relative to ideal clear-sky conditions. This index enables physics-informed DNI estimates when combined with clear-sky DNI (${DNI}_{cs}\left(t\right)$). The method for calculating clear-sky DNI can be found in Section 2.2.3.

In summary, the architecture of the motion model (as shown in Figure 3) is described as follows:

• Input Layer: Two 3-channel images (ROI: 3 × 88 × 88; downsampled all-sky image: 3 × 88 × 88).
• Fusion Layer: Concatenated into a 6-channel tensor (6 × 88 × 88) to preserve local (ROI) and global (all-sky) cloud features.
• Convolutional Blocks: 4 layers (kernel sizes: 22 × 22, 11 × 11, 5 × 5, 2 × 2), each followed by ReLU activation and 2 × 2 max-pooling to extract hierarchical features.
• Fully Connected Layers: Two dense layers (640 → 500 neurons) for feature integration.
• Output Layer: Scalar clear-sky index ($k_{cs}\left(t\right)$), with final DNI forecast computed as $k_{cs}\left(t\right)·{DNI}_{cs}\left(t\right)$.

By leveraging convolutional operations for localized feature detection across fused sky imagery and deep non-linear regression, the motion model effectively correlates transient cloud structures with rapid irradiance fluctuations, providing robust short-term DNI forecasts critical for solar energy grid integration.

2.4.4. Dataset Partition

The training/validation set in this study comprises all-sky imaging and DNI data from 1 August 2023 to 20 August 2024, while the test set encompasses corresponding data from 21 August 2024 to 31 June 2025. For model evaluation, we implemented a 5-fold cross-validation methodology. This technique partitions the original dataset into five mutually exclusive and equally sized temporal subsets (folds) based on daily randomization. One subset was designated as the validation set, with the remaining four subsets combined to form the training set. The process was systematically repeated five times, ensuring each subset served as the validation set exactly once.

2.5. Step-by-Step Description of the Proposed Method

To better illustrate our proposed method, we provide a detailed step-by-step description in Section 8 of the Supplementary Materials.

3. Results

The normalized Mean Absolute Error (nMAE) and the normalized Root Mean Squared Error (nRMSE) are calculated as follows:

$$nMAE={\displaystyle \frac{\frac{1}{n}\sum _{t=1}^n\left|DNI\left(t\right)-DN{I}_{pred}\left(t\right)\right|}{\frac{1}{n}\sum _{t=1}^{n}DNI\left(t\right)}},$$

(11)

$$nRMSE={\displaystyle \frac{\sqrt{\frac{1}{n}\sum _{t=1}^n(DNI\left(t\right)-DN{I}_{pred}\left(t\right){)}^2}}{\frac{1}{n}\sum _{t=1}^{n}DNI\left(t\right)}},$$

(12)

where $n$ is the number of observations, $DNI\left(t\right)$ is the measured DNI, and $DN{I}_{pred}\left(t\right)$ denotes the forecasted DNI.

In this work, we classify the weather into two categories based on whether the maximum cloud pixel velocity magnitude, $\left|v_{\mathrm{m}\mathrm{a}\mathrm{x}}\right|$, exceeds 5 pixels per minute. $\left|v_{\mathrm{m}\mathrm{a}\mathrm{x}}\right|<5$ generally corresponds to cloud-free (clear sky) or overcast skies (uniform cloud cover). These generally represent homogeneous sky states, where irradiance changes are dominated by the sun’s position rather than cloud movement, potentially leading to higher predictability. $\left|v_{\mathrm{m}\mathrm{a}\mathrm{x}}\right|\ge 5$ indicates cloudy skies with rapid cloud movement and high dynamics, which typically pose greater forecasting challenges due to increased complexity and uncertainty in cloud advection and evolution. In Karout et al.’s work [16], days are classified using the following criteria:

• Clear-sky situation: the measured DNI remains within 10% of the clear-sky DNI value for 90% of the day;
• Overcast situation: the measured DNI stays below 60% of the clear-sky DNI value for 90% of the day, with a mean DNI variation under 20 W/m²;
• Mixed situation: all other cases; note that such days may include periods of both clear-sky and overcast conditions.

To compare with the work of Karout et al., besides our own classification, we also categorized the data in Karout et al.’s manner.

In Figure 4, we present the results on the test set under different classifications. As noted by Yang et al. [41], nRMSE and RMSE can yield contradictory conclusions under certain conditions due to variations in DNI averages; therefore, reporting the DNI averages is essential. The mean DNI values for the five cases—namely, the entire test dataset, scenarios with $\left|v_{\mathrm{m}\mathrm{a}\mathrm{x}}\right|<5$, scenarios with $\left|v_{\mathrm{m}\mathrm{a}\mathrm{x}}\right|\ge 5$, clear-sky/overcast conditions, and mixed conditions—are 425.8, 486.9, 312.2, 603.1, and 345.2 W/m², respectively.

The minimum nMAE and nRMSE values were observed under clear and overcast sky conditions classified using Karout et al.’s criteria. The nMAE monotonically increases from 1.3% at H = 5 min to 3.1% at H = 30 min, while the nRMSE monotonically increases from 3.1% at H = 5 min to 5.0% at H = 30 min. In the study by Karout et al. [16], the nRMSE reached approximately 15% at H = 15 min for clear and overcast days, whereas the corresponding nRMSE for our proposed model was approximately 3.9%. These clear and overcast conditions correspond to the $\left|v_{\mathrm{m}\mathrm{a}\mathrm{x}}\right|<5$ scenario in our model, where the clear-sky DNI model is directly applied for clear-sky conditions, and the persistence model is employed for overcast conditions. During clear-sky periods, the model effectively performs clear-sky DNI prediction, while the persistence model is justified for overcast days due to minimal DNI fluctuations. The results demonstrate that this straightforward approach yields notably favorable performance.

The second smallest nMAE and nRMSE values in Figure 4 correspond to the $\left|v_{\mathrm{m}\mathrm{a}\mathrm{x}}\right|<5$ case in our classification. This scenario differs from clear or cloudy days categorized by daily conditions in that it is classified based on maximum cloud pixel speed, encompassing instances with cloud presence but minimal cloud movement. For the $\left|v_{\mathrm{m}\mathrm{a}\mathrm{x}}\right|<5$ case, the nMAE monotonically increases from 3.9% at H = 5 min to 9.3% at H = 30 min, while the nRMSE monotonically increases from 10.4% at H = 5 min to 19.7% at H = 30 min.

The nMAE and nRMSE values for all of the test data monotonically increase from 9.1% and 18.4% at H = 5 min to 16.2% and 28.1% at H = 30 min, respectively. The nMAE and nRMSE values at H = 10 min using our proposed model are 10.8% and 20.7%, respectively, which outperform models in recent studies [13,14,15].

In Figure 4, the mixed scenario exhibits the second-highest nMAE and nRMSE values, increasing from 14.7% and 26.6% at H = 5 min to 25.9% and 40.8% at H = 30 min, respectively.

As shown in Figure 4, the values of nMAE and nRMSE are highest for cases where $\left|v_{\mathrm{m}\mathrm{a}\mathrm{x}}\right|\ge 5$. The larger nMAE and nRMSE values observed under $\left|v_{\mathrm{m}\mathrm{a}\mathrm{x}}\right|\ge 5$ conditions compared to mixed conditions occur because $\left|v_{\mathrm{m}\mathrm{a}\mathrm{x}}\right|\ge 5$ represents cloudy conditions with dramatic DNI fluctuations. In contrast, the mixed condition groups data by day, inevitably encompassing periods with minimal variability (cloud-free and overcast DNI conditions), which are relatively easier to predict.

In addition to the nMAE and nRMSE for DNI prediction accuracy, the model’s capability to forecast irradiance ramps is assessed using the Ramp Detection Index (RDI) [42]. The RDI specifically evaluates a model’s performance in identifying significant changes in DNI.

The process begins by calculating the ramp magnitude ($RM\left(t\right)$) at time t:

$$RM\left(t\right)={\displaystyle \frac{\left|DNI\left(t\right)-DNI\left(t+H\right)\right|}{{\mathrm{D}\mathrm{N}\mathrm{I}}_{\mathrm{c}\mathrm{s}}\left(t\right)}},$$

(13)

where ${\mathrm{D}\mathrm{N}\mathrm{I}}_{\mathrm{c}\mathrm{s}}\left(t\right)$ represents the estimated clear-sky DNI. Typically, major ramps are defined as $RM\left(t\right)$ > 0.5, while moderate ramps fall within 0.3 < $RM\left(t\right)$ < 0.5.

In an RM range, e.g., 0.3 < $RM\left(t\right)$ < 0.5, a successful ramp detection, or a “hit”, occurs when the following condition is met:

$$\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}(DNI\left(t\right)-DN{I}_{pred}\left(t+H\right))=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}(DNI\left(t\right)-DNI\left(t+H\right)),$$

(14)

where $DNI\left(t+H\right)$ and $DN{I}_{pred}\left(t+H\right)$ denote the measured and forecast DNI at time $t+H$, respectively. If the condition is not satisfied, we call it a “miss”.

The RDI itself is then computed as the following ratio:

$$RDI={\displaystyle \frac{N_{hit}}{N_{hit}+N_{miss}}},$$

(15)

where $N_{hit}$ and $N_{miss}$ represent the total counts of hits and misses, respectively.

RDIs with the RM ranges 0.1–0.2, 0.2–0.3, 0.3–0.5, and >0.5 for various H values for mixed situations are shown in Figure 5. RDI increases with larger RM, which shows the same tendency as in Chu et al.’s work [42]. This is good news: more accurate RDI predictions are obtained for larger RM sizes, since these cause greater receiver damage. For H = 10 min, RDI monotonically increases from 60.7% (0.1 < $RM\left(t\right)$ < 0.2) to 89.2% ($RM\left(t\right)$ > 0.5), which is better than Chu et al.’s results [42], from 31.0% to 75.1%.

For $RM\left(t\right)$ > 0.15, RDI is about 72.8% at H = 5 min and 74.9% at H = 15 min for mixed situations, respectively, which is shown in Figure 6. In Karout et al.’s work [16], RDI is about 80.0% at H = 5 min and 72.1% at H = 15 min for mixed situations, respectively. When H = 5 min, the RDI we obtained was lower than that reported by Karout et al., whereas at H = 15 min, our RDI was larger than theirs.

4. Discussion

The nRMSE of the hybrid model by Karout et al. is approximately 56.3% at H = 15 min, while our proposed model achieves an nRMSE of approximately 32.2%, as shown in Figure 4, significantly outperforming the former. Similar to the hybrid model by Karout et al., our proposed model also incorporates dense optical flow and ROI. The primary distinctions between our model and theirs are as follows:

• Data Grouping and Training: We partition the data based on whether the maximum cloud pixel velocity $\left|v_{\mathrm{m}\mathrm{a}\mathrm{x}}\right|$ exceeds 5. Only data with $\left|v_{\mathrm{m}\mathrm{a}\mathrm{x}}\right|\ge 5$ are used for training the motion model. In contrast, Karout et al. utilize the mixed situation dataset for training where clear-sky/overcast data with low variability are included.
• Fitting the clear-sky DNI model’s ${\mathit{T}}_{\mathit{L}\mathit{I},\mathit{t}}$parameter per hour is adopted, and the fitted parameter is used to calculate the clear-sky DNIs and then the clear-sky DNI index within the hour period when preparing the training data. When performing DNI prediction on the test set, we adopted the same calculation method for the clear-sky DNI as in Karout et al.’s work [16].
• Region of Interest and Uncertainty Handling: The ROI for our deep learning model (i.e., the motion model) is the same size as the solar region. Uncertainty in the cloud speed derived from dense optical flow is addressed by incorporating the three channels of the scaled, distortion-corrected image into the input tensor. Conversely, Karout et al. mitigate this uncertainty by enlarging the ROI size.

We attribute the primary performance difference to the first two distinctions. The separate training of a model specifically for $\left|v_{\mathrm{m}\mathrm{a}\mathrm{x}}\right|\ge 5$ conditions and its subsequent use for future DNI prediction. Training exclusively on this subset reduces the complexity of information the model needs to process. The model only needs to focus on potential occlusion sizes within the ROI and the overall cloud distribution in the image.

The method referred to in the second distinction reduces the error of the DNI clear-sky index in the training set, thereby improving the accuracy of the motion model in predicting the DNI clear-sky index. We believe it contributes to the final improvement.

The third distinction (ROI size vs. image channel inclusion) may have a larger impact when significant changes occur within the ROI cloud, such as cloud emergence or dissipation. However, compared to the complexity of Karout et al.’s hybrid model, we consider the influence of this difference on overall performance to be relatively minor.

As referred in Section 3, when H = 5 min, the RDI we obtained was lower than that reported in Karout et al.’s work. We attribute the lower RDI observed with our model at smaller H values to its weaker capability in detecting clouds near the solar disk. Notably, Karout et al. employed a KNN model incorporating features such as the red-to-blue ratio and cloud edge detection for cloud identification, subsequently inputting the cloud fraction within the ROI into their hybrid model. This methodology effectively enhances cloud detection accuracy in proximity to the sun. In contrast, our model is weak at cloud detection near the sun but readily identifies clouds farther from the solar disk. This explains why our model yields RDI values comparable to Karout et al.’s at H = 15 min. Actually, the sun-region mask in ROI is included to improve the RDI performance at smaller H values. Without the sun-region mask in ROI, the RDI at H = 5 min is only 70%. Further results are given in the Supplementary Materials.

Our still model is essentially a clear-sky DNI model or persistence model. We did not employ CNN and/or RNN to enhance the prediction accuracy of DNI under clear or overcast conditions; as for CSP power plants in recent years, molten salt thermal storage is typically adopted, rendering higher DNI prediction accuracy under clear or overcast conditions meaningless. CSP plants are more concerned with abrupt DNI changes caused by cloud cover, as these sudden DNI fluctuations can subject the receiver to severe thermal shocks, reducing its service life or even directly leading to tube-blocking incidents. Therefore, our model focuses more on DNI prediction under partly cloudy conditions. In fact, our model demonstrates excellent DNI prediction accuracy under partly cloudy skies, and when H exceeds 15 min, the RDI also surpasses that of models reported in previous works. This is crucial for improving the efficiency and stability of CSP systems.

Although the proposed model shows good performance in most cases, the MAE can increase up to 45% under dynamically changing weather conditions at H = 30 min, and the RDI at H = 5 min (72.8%) is lower than Karout et al.’s (80%). Such conditions are particularly critical for accurate power production estimation. To further enhance model performance on dynamic cloud days ($\left|v_{\mathrm{m}\mathrm{a}\mathrm{x}}\right|$ ≥ 5 pixels/min), additional improvements are proposed as follows:

• Integrate cloud-type classification: Add an infrared sky image branch to distinguish cloud optical thickness (e.g., thin cirrus vs. thick cumulus), optimizing radiation attenuation modeling for different cloud types [43,44].
• Dynamic ROI adaptation: Replace the fixed 88-pixel ROI with a size/shape-adaptive ROI, adjusted based on cloud movement direction and velocity to better capture sun-obscuring cloud segments.

In future work, we will focus on improving the RDI when H < 15 min and introducing cloud identification near the sun into the model may effectively achieve this. Additionally, the current model is still limited to single-point DNI prediction. By utilizing several all-sky cameras and solar radiometers to predict cloud-shaded areas and to achieve DNI mapping across the entire mirror field, targeted local control of cloud-shaded heliostats can be realized, which is also a highly meaningful direction.

5. Conclusions

This study proposes a hybrid short-term Direct Normal Irradiance (DNI) prediction model for Concentrated Solar Power (CSP) systems, which adaptively adopts two sub-models according to the maximum cloud pixel velocity calculated by the Farnebäck dense optical flow algorithm. Specifically, a rule-based still model (integrating the clear-sky DNI model and persistence model) is applied when the maximum cloud pixel velocity is less than 5 pixels per minute, corresponding to clear or overcast sky conditions; otherwise, a CNN-based motion model is used for cloudy conditions with dynamic cloud movement.

Experimental validation on the actual dataset from the Hami Pilot CSP plant demonstrates that the proposed model achieves superior prediction accuracy across diverse weather conditions, and its performance shows a monotonic decline with the increase in forecast horizon. For the entire test dataset, the normalized Mean Absolute Error (nMAE) and normalized Root Mean Squared Error (nRMSE) rise steadily from 9.1% and 18.4% at the 5 min forecast horizon to 16.2% and 28.1% at the 30 min horizon, respectively. In the mixed weather scenario (the main application scenario of the model), the nMAE and nRMSE increase from 14.7% and 26.6% at 5 min to 25.9% and 40.8% at 30 min, and the Ramp Detection Index (RDI) for major irradiance ramps (RM > 0.5) at the 10 min horizon reaches 89.2%, which is significantly higher than the results of the existing Chu et al. model (75.1%). Compared with the state-of-the-art hybrid model proposed by Karout et al., the proposed model achieves a substantial performance improvement, with the nRMSE at the 15 min horizon reduced from 56.3% to 32.2%.

The primary innovative contributions of this study are reflected in two aspects, and both contributions effectively address the key limitations of traditional DNI prediction models: First, a cloud velocity-based data partitioning strategy is proposed, where only the dataset with maximum cloud pixel velocity ≥5 pixels per minute is used to train the motion model. This strategy avoids the introduction of irrelevant noise from low-dynamics clear/overcast data in model training, reduces the complexity of model feature learning, and significantly improves the prediction accuracy for dynamic cloudy conditions. Second, an hourly parameter fitting method for the clear-sky DNI model is adopted in the training stage, which effectively reduces the calculation error of the DNI clear-sky index and further improves the prediction accuracy of the motion model for the clear-sky index, laying a foundation for the final high-precision DNI prediction.

Despite the excellent overall performance, the proposed model still has certain limitations in practical applications. On the one hand, the dense optical flow estimation algorithm is sensitive to lighting variations and image noise, which may affect the accuracy of cloud motion detection. On the other hand, the model has a relatively weak ability to detect clouds near the solar disk, which results in a lower RDI (72.8%) at the 5 min short forecast horizon compared with Karout et al.’s model (80%). These limitations lead to a significant increase in the nMAE (up to 45%) for the 30 min forecast under dynamically changing cloudy conditions, which is the key direction for subsequent model improvement.

As a high-precision short-term DNI prediction scheme tailored for CSP systems, the proposed model effectively makes up for the deficiency of traditional models in predicting irradiance ramps caused by cloud movement, and its core performance advantages are highly consistent with the actual operational requirements of modern CSP plants equipped with molten salt thermal storage systems. The model can accurately predict the sudden DNI fluctuations caused by dynamic cloud motion, which is of great practical significance for mitigating the thermal shock of the central receiver of CSP systems, reducing the aging of receiver components and the risk of unplanned shutdowns, and further improving the operational stability, efficiency, and overall market competitiveness of CSP technologies. In future research, we will focus on improving the RDI of the model for short forecast horizons (H < 15 min) by introducing solar-near cloud identification technology, and further explore the multi-point DNI mapping prediction based on multiple all-sky cameras to realize the targeted local control of heliostats in CSP plants.