Short-Term Forecasts of Energy Generation in a Solar Power Plant Using Various Machine Learning Models, along with Ensemble and Hybrid Methods

Abstract

High-quality short-term forecasts of electrical energy generation in solar power plants are crucial in the dynamically developing sector of renewable power generation. This article addresses the issue of selecting appropriate (preferred) methods for forecasting energy generation from a solar power plant within a 15 min time horizon. The effectiveness of various machine learning methods was verified. Additionally, the effectiveness of proprietary ensemble and hybrid methods was proposed and examined. The research also aimed to determine the appropriate sets of input variables for the predictive models. To enhance the performance of the predictive models, proprietary additional input variables (feature engineering) were constructed. The significance of individual input variables was examined depending on the predictive model used. This article concludes with findings and recommendations regarding the preferred predictive methods.

1. Introduction

In recent years, the literature associated with solar panels has covered a broad spectrum of topics. After narrowing them to forecasts with short horizons and filtering out works containing only weather prediction, the topic range was narrowed down to a few main themes. While all of the works also contained PV energy forecasts, the presented categories can mostly be thought of as the main differing factors.

The first category of works focused on applying PV forecasts for industrial and household on-site applications. Wang et al. [1] explored coupling between prosumer PV and electricity demand habits of PV-supplied building users. Variants of principal component analysis and Maximum Likelihood Estimation were used along with net data to obtain a forecast of PV generation and the residual electricity demand. On the other hand, Awais et al. [2] and Huang et al. [3] proposed their prediction modus operandi for irrigation systems and 5G base stations, respectively. The aim of their research was to facilitate the selection of proper operations for industrial, PV-supplied equipment.

Meta-analyses constituted the second category of works and reviewed different approaches to forecasting. Bazionis et al. [4] classified approaches with respect to climatic conditions, forecasting horizons, time resolution, and errors of different methodologies. The authors also determined how common the appearance of given forecasting inputs was. Tsai et al. [5] compared and contrasted the pros and cons of particular approaches. Additionally, distributions of error methods, metrics, and input types were shown. Apart from their own study categorization, Tawn and Browell [6] compared study designs and created recommendations for reproducible, high-quality forecasts. The mentioned works show that ANNs are extremely relevant for forecasting, with LSTM and CNNs being commonly used. Additionally, while weather measurements and NWP are frequently used (50% and 25% respective frequency of appearance in the work of Tsai et al. [5]), satellite cloud images and sky images appeared in ca. 11% of the works, which can be seen as a possible beginning of a new trend.

The third category of works dealt with comparative analyses. Campos et al. [7] focused on different RNNs, while Piotrowski et al. [8] proposed a broader spectrum of methods. At the same time, Etxegarai et al. [9] used only a physical model for energy forecasting, but compared LSTM with CNN for intra-hour solar radiation predictions. This approach seems particularly cost-effective for obtaining satisfactory results with limited data or weather data with only hourly resolution.

Spatial pattern processing was certainly the most abundant among the described themes. In this category of works, the common denominator was usually graph-based neural networks combined with LSTM. The work by Liao et al. [10] can be seen as a base, reference scenario for this approach. The authors tested several single ANN methods and compared the results with a hybrid GCN-LSTM model for multiple forecasting horizons. Meanwhile, Yue et al. [11] constructed their experiment differently. First, the authors performed PV farm clustering followed by LSTM-based time series embedding. Next, they extracted spatial patterns with GCN to finally predict energy with MLP. While both of the works used additional predictors, Zhang et al. [12] targeted their research at enhancing the forecasting capability of bare GCN. For that purpose, the graph connectivity index was used to achieve optimal graph structures. Wang et al. [13] dealt with associated problems and introduced a temporal attention mechanism to augment GCN’s ability to process directed graphs. In contrast to the previous works, Bai et al. [14] proposed an entirely different approach using GGRU. The authors benchmarked the results with an extensive set of state-of-the-art methods, albeit without GCN. Hence, their results cannot be compared to previous works. In some papers, spatial pattern processing was performed without neural networks [15,16]. Just like in the work of Yue et al. [11], Sheng et al. [15] performed clustering analysis first. For prediction purposes, however, the authors used PSO-optimized SVR. Conversely, Wang et al. [16] created a switchable model with switch conditions based on the farm cross-correlation function, cloud motion direction, and physical distance direction of two plants. One of the models used satellite-image-based clusters, while the other used neighboring farm data and cloud information.

Instead of capturing interconnections between spatially distributed farms, works discussing general data extraction (fifth category) focused more on mining into existing data to obtain separate input variables and patterns instead of overlapping ones. For that purpose, Liu et al. [17] used two variants of mode decomposition and Rai et al. [18] proposed Autoencoder instead, while Agga et al. [19] and Liu et al. [20] used CNN. Liu et al. additionally used Synchrosqueezing Wavelet Transform to denoise the data.

The use of ensemble models (sixth category of works) allows for more robust and accurate forecasts; however, various ensemble component integration strategies can be applied to obtain an output from the ensemble. Although the easiest strategy would be the use of a simple average, this approach does not take into account the different contributions that ensemble components bring to the table. More sophisticated strategies were proposed by Guo et al. [21], Sarmas et al. [22] and Dudek [23]. LSTM, SVR, and an adaptive ensemble method with stochastic configuration networks, respectively, were shown to be promising integration strategies. The second solution could be potentially the easiest to replicate due to multiple existing implementations of the SVR model.

Works pertaining to data clusters present a seemingly already standardized approach. Researchers have divided weather conditions to obtain weather profiles (or data clusters) corresponding to different types of day, namely sunny, cloudy, rainy, and extreme/overcast. Liu [24] and Huang et al. [3] used the K-means method to obtain the mentioned profiles, while Zhang [25] used a method based on the deviation ratio $\mathsf{\beta }$.

The rest of the analyzed works fit the best into the general “forecast with machine learning” category. Elsaraiti et al. [26] tested the accuracy of LSTM and Chai et al. [27] proposed an approach with WNN optimized with PSO. Last but not least, Mitrentsis et al. [28] presented probabilistic forecasting using NGboost. The work additionally analyzed how confident one can be with a given forecast and used SHAP to fully investigate why a prediction was made.

Our work fits into multiple presented categories. First of all, created forecasts are used not only for the sole sake of forecasting, but are to be used further in the optimization of associated industrial plants. Second, a comparison of different methods and approaches is provided, along with strategies for the creation of ensemble models. Next, a robust accuracy assessment method is applied, and the results of forecasts are compared and contrasted in terms of their sensitivity to particular input variables. Finally, guidelines are made to determine which approach should be used under different present states one may deal with in industrial applications. All in all, the presented methodology aims to create easy-to-implement, transferable solutions as a part of the International DIEGO project (Digital Energy Path for Planning and Operation of sustainable grid, products and society), with increased industrial energy efficiency in mind [29]. It is noteworthy that in our work, we deal with one forecast horizon and one industrial plant. A natural future extension of our framework could be the implementation of federated learning for virtual power plants due to data-sharing security [30] and multi-horizon forecasting [31] to facilitate more robust decision-making systems.

The conducted research was carried out step-by-step, starting from the acquisition of raw data from the photovoltaic system located on the premises of the industrial plant, and ending with recommendations regarding preferred models for practical implementation in the industrial plant. Figure 1 shows the successive steps of the process, divided into “data” and “models”.

2. Data and Forecasting Methods

Raw data were obtained from the measurement system of an industrial plant in which PV was installed. The plant was located in Poland and its measuring system allowed for storing both weather and electrical measurements. Net radiation, air temperature, wind speed, total active power, and total power yields were the variables acquired from the system. The data spanned from 1 January 2022 until the end of 2022. In total, one year of data with intra-minute resolution was acquired.

Real-world raw measurements can rarely be acquired without applying problem-solving procedures. Typical problems associated with time series data usually concern the detection of time changes in data, detection of irregular time structures and their restructuring, filtering out irrelevant data, missing data detection and imputation, etc. Since the final forecast results depend strongly on data quality, lack of use of those procedures can severely impact final forecasting model accuracy. The application of data-handling procedures will be described in Section 2.1.1.

2.1. Data

2.1.1. Preprocessing of “Raw” PV Generation of Electrical Energy

First, it was noted that received generation data had an ever-increasing amount of energy accumulated from the installation of an energy meter until the moment of observation. This data structure had to be transformed into energy accumulated over only the last known observation period. As the second step, we analyzed whether sudden jumps in time appear, especially around the timestamps corresponding to daylight savings time. Since no sudden jumps were detected, it was determined that data points came from uniform time zone measurements and time zone transformation was not required. It is noteworthy that prior to further cleaning, the timestamps in the data had irregular time resolution, often higher than one minute; hence, singular missing samples did not influence the time zone consistency analysis. Next, since the data had varying intra-minute sampling times, samples were grouped by their corresponding timestamp (by timestamp with the same time properties except seconds), aggregated, and then regularized into time series with a uniform set of timestamps with a uniform 1 min resolution. Regularization in this case means merging data by all timestamps with a 1 min resolution for the entire year 2022 in order to properly address missing samples non-detectable prior to regularization. In general, before imputation, data clearing had to be conducted first. To do that, statistical metrics were calculated and time series plots were made as a means of data clearing quality control. These measures allowed us to filter out improper net radiations exceeding 1000 W/m², noticeable in Figure 2.

Additionally, total active power and its yields (energies) were visualized to catch the eventual appearance of unusual changes in time series behavior which would indicate a need for further clearing. Those plots in a general view and in fragments are presented in Figure 3 and Figure 4. Since data anonymity had to be preserved, no exact values are marked on the vertical axes.

While for the active power, no improper values were found, for its corresponding energies, sudden spikes (noticeable in Figure 4) were identified and filtered out. Next, data completeness was analyzed, with data completeness matrix created in missingno package is presented in Figure 5.

In the completeness matrix, a visible lack of data could be observed for air temperature and wind speed. Since averages over periods with lower frequency should be more robust, the imputation procedure was to be completed after data aggregation into 15 min periods. After data clearing and changing to a 15 min resolution, a data completion algorithm was established. Wind speed and air temperature values were filled forwards, then backwards to deal with singular empty records. The rest of the empty records were filled with daily medians of the analyzed parameters.

2.1.2. Statistical Analysis of Time Series of PV Generation of Electrical Energy and Meteorological Measurement Data (15 min Periods)

The time series data (PV generation and meteorological data) span the entire year 2022. All four time series have been limited to data in 15 min intervals from sunrise to sunset (during other periods, the generation is zero, making them irrelevant for forecasting models and introducing unnecessary “noise”). Electricity generation from a PV system has diurnal variability. The greatest generation of electricity occurs around midday. Figure 6 shows the daily variability of electricity generation in August 2022 (summer month), with the standard deviation and median for each hour. The distribution of standard deviation values coincides in practice with the distribution of electricity generation values however there are some differences—a flattening of the distribution of standard deviation values in the period of the middle of the day despite quite big changes in generation values.

Figure 7 shows the scatter plot of electrical energy generation values and solar irradiance values. The data in the XY coordinate system are fitted with a curve using a weighted least squares smoothing procedure, where the influence of points decreases with their horizontal distance from a given point on the curve. A slight nonlinearity is observed on the fitting curve for the highest values of electricity generation; the growth dynamics of generation decrease with increasing solar irradiance. This effect is likely due to the decreased efficiency of photovoltaic panels under very high air temperatures.

2.1.3. Development of Input Variable SETS for Forecasting Models Including Feature Engineering

In the first step, an analysis was conducted to verify which lagged values of the forecasted time series of electricity generation could potentially serve as valuable input data for predictive models. For this purpose, Pearson linear autocorrelation values for the electricity generation time series up to 10 days backward (960 lagged 15 min values) were calculated. Figure 8 shows the autocorrelation function (ACF) of electricity generation. The value of the Pearson linear correlation coefficient for a lag of exactly 1 day (96 lags) is 0.770, which is greater than the correlation coefficient of lag 6 (lag 6 has a correlation coefficient of 0.788, while lag 7 has a value of 0.7526). For lags that are multiples of 96, the correlation coefficient decreases very slowly, and for a lag of exactly 10 days, it is 0.672. In conclusion, using the last 6 lagged values and several lags that are multiples of 96 (exactly the same periods in previous days) as potential input data for predictive models is justified based on the correlation magnitude.

Next, weighted averaging of the time series of electricity generation values was performed to reduce the random component in the data. The selected past values of time series transformed in such a manner may be valuable inputs and possibly even replace the past values of the forecasted time series as input data in the forecasting model. The smoothed time series values of power generation were calculated using Formula (1). The value of the R coefficient between energy generation values in period T (output data) and smoothed energy generation values in period T-1 (input data) is very high (0.9275) and greater than that for energy generation in period T-1 (0.9246). This indicates the potentially significant importance of the input variable of smoothed energy generation for the forecasting model.

$${EG}_t^{smoothed}={EG}_{t-1}·w_{t-1}+{EG}_{t-2}·w_{t-2}+{EG}_{t-3}·w_{t-3},$$

(1)

where ${EG}_t^{smoothed}$ is the smoothed value of electricity generation for period t, ${EG}_{t-k}$ is the value of electricity generation for period t-k, $w_{t-1}=0.716$, $w_{t-2}=0.128$, and $w_{t-3}=0.06$.

In order to potentially enhance the effectiveness of the predictive models, two additional markers (input data) were introduced. In the case of 15 min intervals where solar irradiance naturally increases (until noon), an additional marker called “rising solar irradiance” takes a value of 1. The model can potentially benefit from the additional information that subsequent intervals are expected to experience an increase in solar irradiance (assuming no cloud cover). Conversely, for 15 min intervals after 12:00 p.m., an additional marker called “declining solar irradiance” equals −1. The model can potentially benefit from the additional information that subsequent intervals are expected to see a decrease in solar irradiance (assuming no cloud cover).

Due to the pronounced seasonality of electricity generation in the PV system (higher generation in summer months compared to winter months) and the strong daily variability of the electricity generation process, two additional markers, “hour” and “month,” representing daily variability and seasonality, respectively, were introduced. It is believed that these additional input data for predictive models may provide information that can assist in effective forecasting.

Table 1. All selected potential input and output data and codes.

Output data description	Code
Generation in period T [p.u.]	EG(T)
Input data description	Code
Month	Month
Hour	Hour
Rising solar irradiance	R_SI
Declining solar irradiance	D_SI
Smoothed generation in period T-1 [p.u.]	SEG(T-1)
Generation in period T-n, n = 1, 2…6, 96, 192 [p.u.]	EG(T-n)
Solar irradiance T-n, n = 1, 2…6, 96, 192 [W/m2]	SI(T-n)
Air temperature T-1 [°C]	AT(T-1)
Air temperature T-2 [°C]	AT(T-2)
Wind speed T-1 [m/s]	WS(T-1)

contains a list of all prediction inputs (24) and outputs (1) and their corresponding codes. Summarizing the analysis of nonlinearity between input and output variables, the following conclusions can be formulated:

• The most important input variables are recent lagged values of electricity generation and solar irradiance. They exhibit almost linear dependence with the output data EG(T).
• The relationship between EG(T) and the lagged value of one day, EG(T-96), is nonlinear in nature.
• Air temperature and wind speed show a relatively small correlation with EG(T), and their relationship is nonlinear.
• When using input variables with a nonlinear relationship to the output, it seems appropriate to use models capable of modeling nonlinearity; however, for simple models with a limited number of inputs, which exhibit either linear or nearly linear relationships with the output, one can use predictive models that model linear dependencies, such as ARIMA or Multiple Linear Regression models.

In the next step, an importance analysis was performed for input data. The analysis concerned MLP, MLR, RF, and XGBOOST models for which input importance ranking was created. For the MLP model, sensitivity analysis involves varying input variables by changing the values of one input variable at a time while keeping other inputs constant. In this analysis, each input variable’s values are replaced with a constant value, specifically the mean value of all samples. The forecast quality is then assessed after this replacement, and the ratio of the forecast error with replacement to the forecast error without replacement is calculated. A higher ratio indicates a greater impact of the input variable on forecast quality. For instance, a ratio of 2 signifies that the forecast error has doubled compared to the original error. Conversely, a ratio below 1 suggests that removing this input variable would likely improve forecast accuracy, as the forecast error is lower without it.

In the case of RF, the importance of a given feature is the sum of the impurity reduction achieved at each node where the feature was used for splitting, weighted by the number of samples passing through that node.

In the case of XGBoost, the importance of a given feature is determined by how frequently the feature is used for splitting in the trees and how much it improves the model’s performance. This is measured as the average reduction in loss (mean squared error) at the locations where the feature was used.

The MLR model was used with backward stepwise elimination of input data. The algorithm eliminated input data in 12 steps. In the final model, the 13 most important input data were retained for the Multiple Linear Regression model. The observed power values (alpha = 0.05) were examined for the model obtained in step 12 (13 input data). The observed power values were adopted as a measure of importance for the 13 input data (the higher the observed power value, the greater the importance of the respective input data).

For each method, inputs were sorted by descending importance and given points corresponding to how many places from last a given variable was in the ranking. The final position in the ranking was based on points summed over all methods and represented overall input variable importance. Based on the ranking, it can be concluded that the most important inputs are electricity generation and solar irradiance from the last few 15 min periods before the forecast period. Additionally, smoothed generation in period T-1 is also among the most important input data. On the other hand, the least valuable input data are wind speed and the seasonality marker (Month). In Figure 9, the final ranking of the importance of input data for linear and nonlinear models is presented.

Based on the results obtained from the analyses, six different, prediction-applicable sets of input variables were expertly proposed. Sets ranged from the most comprehensive version—all inputs—to the simplest version using only the latest value of the forecasted process.

Table 2. Sets of input data selected for forecasting methods.

Code of Set	Codes of Input Data and Additional Comments
SET 1 (24 inputs)	Month, Hour, R_SI, D_SI, SEG(T-1), EG(T-1), EG(T-2), EG(T-3), EG(T-4), EG(T-5), EG(T-6), EG(T-96), EG(T-192), SI(T-1), SI(T-2), SI(T-3), SI(T-4), SI(T-5), SI(T-6), SI(T-96), SI(T-192), AT(T-1), AT(T-2), WS(T-1). All available/created input data including endogenous variables, exogenous variables, seasonality markers, daily variability markers, and process trend markers (increasing/decreasing)
SET 2 (12 inputs)	SEG(T-1), EG(T-1), EG(T-2), EG(T-3), EG(T-4), EG(T-5), EG(T-6), EG(T-96), SI(T-1), SI(T-2), SI(T-3), SI(T-4)—12 highest ranked input data (from 24 input data) based on the final balancing ranking of the importance of input data
SET 3 (13 inputs)	Month, Hour, R_SI, D_SI, SEG(T-1), EG(T-1), EG(T-2), EG(T-3), EG(T-4), EG(T-5), EG(T-6), EG(T-96), EG(T-192). Only endogenous variables and seasonality markers, daily variability markers, and process trend markers (increasing/decreasing)
SET 4 (9 inputs)	SEG(T-1), EG(T-1), EG(T-2), EG(T-3), EG(T-4), EG(T-5), EG(T-6), EG(T-96), EG(T-192) Only endogenous variables without markers
SET 5 (1 input)	EG(T-1)
SET 6 (p inputs)	EG(T-1), …, EG(T-p); for ARIMA models, p is tested from 2 to 8

shows proposed datasets that will be applied to forecasts using various methods.

2.1.4. Dataset Division

In order to divide the dataset into training, validation, and test subsets, the full dataset was divided into 4 climatic seasons in full 2022 year. One last week of each season was labeled as test data, while the rest of the data were treated as a combined training/validation dataset. For this combined dataset, one last week of each season was taken as validation data. Overall, data were divided into training, validation, and test datasets with an 83.4–16.6% proportion, where validation data constituted 20.7% of the samples in the training/validation dataset.

2.2. Forecasting Methods

In order to obtain the broadest perspective on the qualitative potential of various forecasting methods for the task of energy generation forecasting in a PV system within a 15 min horizon, expert-selected methods of varying complexity were chosen. This allows us to verify the necessity of constructing highly complex models and assess how complex input data should be used in given forecasting problems. The following classes of methods are tested: single methods (linear forecasting models, nonlinear forecasting models) and complex methods—ensemble and hybrid models (including homogenous and heterogenous models). Figure 10 shows the classification of the applied forecasting methods.

The tested and tuned single-type forecasting models are as follows: persistence model (NAIVE), Autoregressive Integrated Moving Average (ARIMA), Multiple Linear Regression (MLR), K-Nearest Neighbors (KNN), Support Vector Regression (SVR), Long Short-Term Memory (LSTM), and Multilayer Perceptron (MLP).

The NAIVE model is a simple forecasting technique that makes predictions based on the assumption that future values will be the same as the most recent observed value [8]. The ARIMA model is a widely used statistical method for time series forecasting. It combines three components: the autoregressive (AR) part, which models the relationship between an observation and a number of lagged observations; the integrated (I) part, which involves differencing the series to make it stationary; and the moving average (MA) part, which models the relationship between an observation and a residual error from a moving average model applied to lagged observations.

The MLR model is a statistical technique used to predict a target variable based on the linear relationships between the target and multiple predictor variables. It extends simple linear regression by incorporating multiple independent variables, allowing it to capture more complex relationships in the data.

The KNN model (version for regression tasks) is a non-parametric algorithm that makes predictions based on the average value of the target variable from the K-nearest training data points to a given test point [32]. Unlike linear models, KNN does not assume a specific form for the relationship between predictors and the target variable; instead, it relies on the similarity between data points.

SVR is a type of regression technique within the Support Vector Machine framework, designed to predict continuous values [8]. SVR aims to find a function that deviates from the actual observed values by no more than a specified margin, known as the epsilon (ε), while also minimizing the model complexity. It operates by mapping input data into a high-dimensional space using a kernel function, where it then constructs a regression function that is as flat as possible within the margin. SVR is effective in handling both linear and nonlinear relationships.

LSTM networks are a type of Recurrent Neural Network (RNN) designed to capture long-range dependencies in sequential data [23]. Unlike traditional RNNs, LSTMs address the problem of vanishing and exploding gradients by using a specialized architecture that includes memory cells and gating mechanisms. These gates—input, output, and forget gates—regulate the flow of information into and out of the memory cells, allowing the network to maintain relevant information over long sequences. LSTMs are particularly effective in time series forecasting.

MLP is a type of feedforward artificial neural network used for supervised learning tasks. It consists of multiple layers of neurons, including an input layer, one or more hidden layers, and an output layer [33]. Each neuron in a layer is fully connected to neurons in the subsequent layer, and the network learns by adjusting the weights of these connections through backpropagation. MLPs are capable of modeling complex, nonlinear relationships.

In

Table 3. Detailed information on single-type forecasting models.

Model Name/ Model Code	Type	Tested Sets of Input Data	Hyperparameters/Parameters Tuned
Persistent model/ NAIVE	Linear	SET5	-
Autoregressive Integrated Moving Average/ ARIMA	Linear	SET6	p—autoregressive order; d—differencing order; q—moving average order; c—constant term (total number of tested model variants: 32)
Multiple Linear Regression/ MLR	Linear	SET1…SET4	β0—constant term; β1, β2, …, βn—coefficients (total number of tested model variants: 8)
Multilayer Perceptron/ MLP	Nonlinear	SET1…SET5	Number of hidden layers and neurons in each layer, activation functions in each layer, optimizer type including optimizer-specific parameters, weight initialization type, number of epochs, batch size (total number of tested model variants: 498)
Long Short-Term Memory/ LSTM	Nonlinear	SET1…SET5	Number of hidden layers and neurons in each layer, activation functions in each layer, optimizer type including optimizer-specific parameters, weight initialization type, early stopping patience, batch size (total number of tested model variants: 240)
Support Vector Regression SVR	Nonlinear	SET1…SET5	Type of kernel; C—punitive term; ϵ—error tolerance (total number of tested model variants: 3400)
K-Nearest Neighbors/ KNN	Nonlinear	SET1…SET5	K—number of neighbors, distance metric (total number of tested model variants: 450)

, there is detailed information on single-type forecasting models including the tuned hyperparameters of models. The total number of tested single-type forecasting model variants is 4628.

In the next step, homogeneous ensemble models were tested, and more advanced heterogeneous ensemble models were proposed and tested. Both classes of ensemble models have advantages as well as certain drawbacks.

The advantages of homogeneous ensemble models include simplicity of implementation and effective performance in the case of models that tend to have high variance (e.g., decision trees). Their drawback can be the potential difficulty in capturing all aspects of the data if all predictors share the same weaknesses. In turn, heterogeneous ensemble models can capture diverse aspects of the data by using predictors of different types and often offer greater flexibility and potentially higher prediction accuracy, especially in complex problems. Their drawbacks include greater complexity of implementation, higher computational resource requirements, and more complex training procedures.

Random Forest (RF) is a homogenous ensemble learning method that creates a collection of decision trees to predict continuous outcomes [33]. Each tree in the forest is trained on a random subset of the training data, and when making predictions, the model averages the outputs of all individual trees to provide the final prediction. This approach helps to capture complex patterns and interactions in the data while reducing overfitting compared to a single decision tree. The averaging process smooths out the predictions, leading to more robust and accurate results for regression tasks.

Extreme Gradient Boost Decision Tree (XGBOOST) is an implementation of gradient boosting that builds a homogenous ensemble of decision trees in a sequential manner, where each new tree corrects the errors made by the previous ones [33]. XGBOOST optimizes both the model’s performance and computational efficiency through techniques like regularization to prevent overfitting, and advanced algorithms for parallelization and tree pruning. The forecast in XGBOOS is a weighted sum of the outputs from all decision trees in the ensemble, where each tree’s contribution is based on the errors it is designed to address.

Table 4. Detailed information on ensemble-type forecasting models with homogenous-type predictors in the ensemble.

Model Name/ Model Code	Operating Method	Ensemble Prediction Calculation	Hyperparameters Tuned
eXtreme Gradient Boost Decision Tree/ XGBOOST	Boosting	Averaging	The number of decision trees, maximal depth of each decision tree, learning rate (total number of tested model variants: 500)
Random Forest/ RF	Bagging	Averaging	The number of randomly chosen input data for each decision tree individually, the number of decision trees, minimum number of samples in a node subject to splitting, the maximum number of levels, the maximum number of nodes, minimum samples per leaf (total number of tested model variants: 192)

shows detailed information on ensemble-type forecasting models (RF and XGBOOST) with homogenous-type predictors (decision trees) in the ensemble. Tested sets of input data for both models: SET1…SET5.

To develop and test more advanced ensemble models with heterogeneous type predictors (each predictor of a different type), a set of models with qualitatively similar results was selected. The selection was made from single models and homogeneous ensemble models. For a given type of model, only the best model was chosen. The error metric for model selection was nRMSE. Four models were finally selected and ordered by the model with the smallest nRMSE error (RF, XGBOOST, MLP, and LSTM). For the purpose of verifying the linear similarity between the errors E of single models, Pearson’s linear correlation coefficients R were calculated for all combinations of pairs of single models of the error E time series forecasts of both models within the training range. Error E is defined as the difference between the actual value and the forecast value. The smaller the value of R, the more theoretically a given pair of single models is suitable for inclusion in a common ensemble due to differences in the time series progression. The smallest R correlations are observed between the XGBOOST and MLP model pair and the XGBOOST and LSTM model pair (a good choice for the ensemble). The largest R correlations are seen between the LSTM and MLP model pair and the RF and XGBOOST model pair (a less favorable choice for the ensemble). In subsequent steps, various ensemble models were constructed using combinations of four models (RF, XGBOOST, MLP, and LSTM). It was assumed that the ensemble model must always include the best single model (RF). Different strategies for constructing ensemble/hybrid models were utilized in the analysis.

Strategy No. 1—Averaging ensemble based on different methods. This integrates the results of selected predictors into the final verdict of the ensemble [33]. The final forecast is defined as the average of the results produced by all different predictors organized in the ensemble.

Strategy No. 2—Weighted averaging ensemble based on different methods. In this strategy, a weighted average is calculated as the final forecast of the ensemble model [33]. It is assumed in this case that single models, which have achieved a lower nRMSE error, carry greater weight in the ensemble model. As a result, each predictor influences the final outcome in proportion to its accuracy. The final prediction result is calculated using Formula (2). The weight values for each predictor are determined by Formula (3).

$${\widehat{y}}_i={\sum }_{j=1}^s{\widehat{y}}_i^j{w}_j,$$

(2)

where i is the prediction point, ${\widehat{y}}_i$ is the final predicted value, ${\widehat{y}}_i^j$ is predicted value by single predictor number j, s is the number of single predictors in the ensemble and $w_j$ is the weight of j-th predictor in the ensemble.

$$w_j={\displaystyle \frac{\frac{1}{{nRMSE}_j}}{\sum _{j=1}^s\frac{1}{{nRMSE}_j}}},$$

(3)

where ${nRMSE}_j$ is the error of the j-th predictor (validation subset).

Strategy No. 3—Ensemble Averaging Without Extremes. This method (developed by the authors of this study [33]) involves the removal of the minimum and maximum forecast from the set of n individual predictors (different types of methods) before each calculation of single final forecasts, using an average of forecasts from n-2 predictors. This should in theory decrease prediction errors. An important condition for including the predictor in the ensemble is mutually independent operation and also similar levels of prediction error. The final prediction result is calculated by Formula (4).

$${\widehat{y}}_i={\displaystyle \frac{1}{n-2}}({\sum }_{j=1}^s{\widehat{y}}_i^j-min\left\{\left.{\widehat{y}}_i^j\right\}-max\left(\left.{\widehat{y}}_i^j\right\})\right.\right.,$$

(4)

where i is the prediction point, ${\widehat{y}}_i$ is the final predicted value, ${\widehat{y}}_i^j$ is the predicted value by a single predictor number j, s is the number of single predictors in the ensemble, and n is the number of predictors in the primary ensemble before the elimination of the outputs of predictors yields extreme forecasts from the set of results.

Strategy No. 4—Hybrid with MLP as “meta-model”—integrates the results of the chosen predictors into the final verdict of the ensemble using the MLP model [33]. Finally, 2 to 4 predictors are chosen for the hybrid model based on the smallest nRMSE errors on the validation subset and predictors of differing types. The MLP integrator uses forecasts from individual predictors as input data and the real values of electric energy generation as an output. The training dataset is used for the training of the MLP integrator and validation dataset for nRMSE control and tuning of hyperparameters. Finally, the evaluation criteria are checked on the test dataset. A general scheme of the hybrid strategy with MLP as a “meta-model” is presented in Figure 11.

Strategy No. 5—Hybrid with two models connected in series. In this strategy, the “meta-model” proposed is RF, which, besides the input data SET1 (all available/created input data), receives an additional input (forecast) from the other model, forming a hybrid model altogether [33]. In such a solution, one model supports another in the forecasting process. A general scheme of the hybrid strategy with two models connected in series is presented in Figure 12.

Table 5. Detailed information on ensemble- and hybrid-type forecasting models with heterogenous-type predictors.

Model Name/ Model Code	Operating Method	Ensemble/Hybrid Prediction Calculation	Tested Sets of Predictors
Averaging ensemble based on different methods/ AVE (predictor 1, predictor 2, …, predictor n)	Combining different architectures	Averaging	(RF, XGBOOST, MLP), (RF, XGBOOST, MLP, LSTM), (RF, XGBOOST, LSTM), (RF, XGBOOST), (RF, MLP), (RF, LSTM)
Weighted averaging ensemble based on different methods/ AVE_W (predictor 1, predictor 2, …, predictor n)	Combining different architectures	Weighted averaging	(RF, XGBOOST, MLP), (RF, XGBOOST, MLP, LSTM), (RF, XGBOOST, LSTM), (RF, XGBOOST), (RF, MLP), (RF, LSTM)
Ensemble Averaging Without Extremes/ AVE_OUT_EXT (predictor 1, predictor 2, …, predictor n)	Combining different architectures	Averaging without extreme forecasts for each prediction	(RF, XGBOOST, MLP), (RF, XGBOOST, LSTM), (RF, XGBOOST, MLP, LSTM)
Hybrid with MLP as “meta-model”/ INT_MLP (predictor 1, predictor 2, …, predictor n)	Stacking (stacked generalization)	Output from MLP “meta-model”	(RF, XGBOOST, MLP), (RF, XGBOOST), (RF, XGBOOST, LSTM, MLP), (RF, XGBOOST, LSTM)
Hybrid with two models connected in series/ predictor 1->predictor 2	Stacking (stacked generalization)	Output from predictor 2 “meta-model”	XGBOOST->RF, MLP->RF, LSTM->RF

shows detailed information on ensemble- and hybrid-type forecasting models with heterogenous-type predictors. Tested sets of input data for ensemble models: SET1…SET5.

Figure 13 presents a summary of the model classes used in this study, including a list of the model names belonging to each class.

3. Results

In order to have a broader view of the quality of individual forecasting models, four evaluation criteria are used, including nRMSE, nMAE, nAPEmax, and nMBE. The nAPEmax and MBE measures are only auxiliary evaluation criteria.

Root Mean Square Error is calculated by Formula (5). The nRMSE measure is typically used for power generation forecasts from RES, including PV systems.

$$nRMSE=\sqrt{{\displaystyle \frac{1}{c_{norm}}}{\displaystyle \frac{1}{n}}{\sum }_{i=1}^n(y_i-{\widehat{y}}_i{)}^2},$$

(5)

where $c_{norm}$ is the normalizing factor (nominal electricity generation in a 15 min period), ${\widehat{y}}_i$ is the predicted value, $y_i$ is the actual value, and $n$ is the number of prediction points.

Normalized Mean Absolute Error is determined by Formula (6). Due to the zero values occurring in the power generation time series, it is impossible to use the popular and recommended measure of the MAPE error.

$$nMAE={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{\displaystyle \frac{1}{c_{norm}}}\left|y_i-{\widehat{y}}_i\right|,$$

(6)

Normalized Maximum Absolute Percentage Error is calculated by Formula (7). The nAPEmax error is the largest partial error of all individual n nAPE errors.

$$APEmax=\underset{i=1,...,n}{max}{\displaystyle \frac{1}{c_{norm}}}\left|y_i-{\widehat{y}}_i\right|·100\%,$$

(7)

Normalized Mean Bias Error (MBE) captures the average bias in the prediction and is defined by Formula (8). The forecasting method underestimates values if nMBE < 0 or overestimates values if nMBE > 0. The MBE of a properly functioning prognostic method should be equal to or very close to zero.

$$nMBE={\sum }_{i=1}^n{\displaystyle \frac{1}{c_{norm}}}\left(y_i-{\widehat{y}}_i\right)$$

(8)

Among the four presented metrics for evaluating forecast quality, the normalized Root Mean Squared Error (nRMSE) was selected as the primary measure for choosing the best predictive models. nRMSE is commonly used as the primary measure for assessing the quality of forecasting models due to its emphasis on evaluating larger values rather than very small ones in the forecasted time series (the measure employs the squared error).

Table 6. Quality ranking of all analyzed models (one model from each category).

Model Variant (Code)	Input Data Code	nRMSE (p.u.)	nMAE (p.u.)	nAPEmax (%)	nMBE (p.u.)
AVE_W(RF,XGBOOST,MLP)	SET1, SET3	0.0210397	0.009777	19.118	−0.000612
AVE(RF,XGBOOST,MLP)	SET1, SET3	0.0210399	0.009776	19.120	−0.000611
AVE_OUT_EXT(RF,XGBOST,MLP)	SET1, SET3	0.0210681	0.009814	19.356	−0.000520
INT_MLP(RF,XGBOOST,MLP)	SET1, SET3	0.0211300	0.009646	19.460	−0.000175
RF	SET1	0.0212316	0.010263	18.800	−0.000711
XGBOOST	SET3	0.0213606	0.009863	19.706	−0.000524
XGBOOST->RF HYBRID	SET1, SET3	0.0213951	0.009498	20.034	−0.000446
MLP	SET3	0.0214949	0.009953	19.356	−0.000599
LSTM	SET3	0.0215378	0.010164	18.563	−0.000436
KNN	SET4	0.0221539	0.009742	19.667	−0.000670
MLR	SET3	0.0222672	0.011174	19.591	−0.000427
ARIMA	SET6	0.0226611	0.011380	19.767	−0.000870
SVR	SET3	0.0228177	0.012782	17.247	−0.000903
NAIVE	SET5	0.0237388	0.009884	20.292	0.000347

presents a quality ranking of all the models analyzed. The results are sorted in ascending order based on the nRMSE error. The best result for each error metric is highlighted in bold, while the worst result is marked in red. When creating the ranking shown in Table 6, one model from each category was selected that had the smallest nRMSE error.

It is noteworthy that the proposed models significantly reduce the magnitude of the nRMSE error more than the nMAE error. The nRMSE error improvement over the naive method (%) depending on the forecasting method is presented in Figure 14. However, Figure 15, Figure 16, Figure 17 and Figure 18 present actual electricity generation values and forecasts by the best model named “Weighted averaging ensemble based on different methods” AVE_W(RF,XGBOOST,MLP) for three consecutive days (only periods from sunrise to sunset) during different seasons.

Based on the results presented in Table 6, several observations can be made:

• The best results (measured by the error metric nRMSE) were achieved by heterogeneous ensemble models. The hybrid model was slightly worse than the two homogeneous ensemble models (RF and XGBOOST). It is worth noting that the differences in quality between these models are minimal.
• Nonlinear single models (KNN and SVR) perform significantly worse in terms of quality compared to ensemble models.
• Particular attention should be given to the very poor performance of the single SVR model, which is worse than the linear single models (MLR and ARIMA).
• Linear single models (MLR and ARIMA) are distinctly inferior in quality compared to nonlinear models.
• It is worth noting that nonlinear single models (MLP and LSTM) offer high quality comparable to that of both homogeneous and heterogeneous ensemble models, while being significantly less complex in their construction.

4. Discussion

In the conducted research, both linear (three different types) and nonlinear predictive methods (six different types) were tested. Additionally, ensemble and hybrid methods were developed.

The best (nRMSE error) ARIMA model has the following hyperparameters: p = 7, d = 0, q = 2. Differencing (hyperparameter d) is not advantageous when the goal is to minimize nRMSE error. It is worth noting that statistical analyses indicated that the time series is non-stationary, hence differencing (hyperparameter d) is justified. However, differencing is advantageous when the goal is to minimize nMAE error. When differencing is applied, the constant term c in the ARIMA model equation is always statistically significant, implying that building a model without the constant term c is unnecessary.

The best MLR model is the model without the constant term c, and the best input dataset is SET(3). The variant with the constant term c was unable to generate forecasts with a value of zero, whereas the variant without the constant term was capable of generating forecasts with a value of zero (when the actual energy generation value was zero). Analyzing the coefficients’ values of the equation in the best MLR model, it can be concluded that the model strongly focused on the input variable SEG(T-1), considering it as the most important.

The best MLP model has 14 neurons in the hidden layer and 13 input data (SET3). The activation functions in the hidden and output layers are hyperbolic tangent and linear, respectively. The utilization of markers (SET3) compared to the variant with the same input data but without markers (SET4) improved the result by 2.6%, indicating that the application of markers is fully justified. In all model variants (different sets of input data), the global sensitivity analysis conducted for the MLP neural network revealed that in each model, the most crucial input data is EG(T-1), with SEG(T-1) taking the second place. Sensitivity coefficients for these input data are considerably higher than for the remaining input data.

The best LSTM model has eight neurons in the hidden layer and 13 input data (SET3). The activation function in the hidden/output layer is hyperbolic tangent/linear. For the early stopping mechanism, 50 epochs resulted in the best results for almost all input datasets.

The best XGBOOST model is built from 86 decision trees with a maximal depth of 3, a learning rate of 0.1, and 13 input variables (SET3). XGBOOST models visibly preferred shallow architecture and slow learning rates.

The best RF model has 300 trees, up to 10 levels in each decision tree, and minimum 10 samples per leaf. The number of randomly chosen input variables for each decision tree is 80% from 24 inputs (dataset SET1). Among the 4 tested sets of input data, SET1, which is the largest dataset (with the lowest nRMSE error), is the best. Therefore, the Random Forest model differs from other machine learning techniques, where SET3 is the best set of input data. The utilization of markers (SET3) compared to the variant with the same input data but without markers (SET4) improved the result by 0.94%, indicating that the application of markers is fully justified.

The best model SVR has a punitive term C equal to 1, an error tolerance ϵ equal to 0.1, a polynomial function as a kernel, and 13 input variables (SET3). It can be noticed that SVR preferred low punitive terms and low tolerance factors.

The best KNN model has 16 neighbors (k) and uses the Manhattan distance metric and nine input variables (SET4). For four out of five sets, the Manhattan distance metric turned out to be the best. It can be noticed that the differences between models for different sets were small, and when compared to SET5 with only one input, the result did not improve much. This can be interpreted as variables from SET5 being the most valuable for the model.

The ensemble model named “Averaging ensemble based on different methods” AVE(RF,XGBOOST,MLP) achieved the smallest nRMSE and nMAE errors. The analysis indicates that higher-quality ensemble models were obtained by using three or four single models in the ensemble. Using only two single models (including the best model RF) is a slightly worse choice. It is worth noting that the nRMSE error difference between the second and third ensemble models in the ranking is very small.

The ensemble model named “Weighted averaging ensemble based on different methods” AVE_W(RF,XGBOOST,MLP) achieved the smallest nRMSE and nMAE errors. The analysis indicates the same conclusions as for the model “Averaging ensemble based on different methods”.

The ensemble model named “Ensemble Averaging Without Extremes“ AVE_OUT_EXT(RF,XGBOOST,MLP) achieved the smallest nRMSE and nMAE errors.

The hybrid model named “Hybrid with MLP as “meta-model”“ INT_MLP(RF,XGBOOST,MLP) achieved the smallest nRMSE error. Compared to the RF model (the best from homogenous-type ensemble models), the percentage decrease in nRMSE error is 0.478%. However, the hybrid model INT_MLP(RF,XGBOOST) achieved the smallest nMAE error from all ensemble and hybrid models. The sensitivity analysis conducted for the best model showed that for the MLP as an integrator, the most important input variable is the forecast from the model XGBOOST. In second place as an input variable is the forecast from the model RF. The MLP integrator has 3 neurons in the hidden layer. The activation function in the hidden and output layers is a hyperbolic tangent and linear function, respectively.

The best model, named “Hybrid with two models connected in series” XGBOOST->RF, has a 9.87% lower nRMSE error in comparison to the NAIVE model. The feature importance analysis showed that the most important input data for the RF “meta-model” is the forecast from the XGBOOST model, with the second most important being EG(T-1).

Insights regarding the comparison of the quality of the proposed ensemble and hybrid models include the following:

• The “Weighted averaging ensemble based on different methods” model is marginally better than the “Averaging ensemble based on different methods” model—the difference is extremely small.
• The best of the “Ensemble Averaging Without Extremes” models has forecast results that are worse than both the best “Weighted averaging ensemble based on different methods” model and the best “Averaging ensemble based on different methods” model.
• All analyzed ensemble models achieved an nRMSE error smaller than the best single models and ensemble with homogenous input data model (RF).
• The smallest nMAE from all analyzed models was achieved by “Hybrid with two models connected in series” (XGBOOST->RF).

5. Conclusions

Based on the obtained results, it is worth noting that nonlinear models considerably outperformed linear models in terms of forecast error magnitude. Combining individual models into an ensemble model is advantageous, although the quality improvement is relatively small. Among the input datasets, SET1 is often the best (all available/created input data), with SET3 being slightly less frequent. It is not possible to definitively determine which set is optimal as it depends on the category of the predictive model.

To provide a broader perspective on the issue of selecting the preferred predictive model and preferred input dataset, the problem was divided into two criteria based on the decision-maker’s needs. Six different input datasets were used in these studies.

I. Selection Criterion: Best forecasting quality regardless of model complexity.

• The preferred model when considering nRMSE error as the most critical quality criterion is AVE_W(RF, XGBOOST, MLP). This model shows an improvement of 11.370% over the naive method and uses SET1 (the most extensive set) and SET3 as input data.
• The preferred model when considering nMAE error as the most critical quality criterion is the XGBOOST->RF HYBRID. This model shows an improvement of 3.901% over the naive method and uses SET1 and SET3 as input data.
• The preferred model when both nRMSE and nMAE errors are considered the most critical quality criteria is INT_MLP(RF, XGBOOST, MLP). This is the most complex of all models and requires the most time to build and finetune hyperparameters. The improvements over the naive method are 10.990% and 2.411%, respectively, using SET1 and SET3 as input data.

II. Selection Criterion: Model simplicity while maintaining good forecasting quality.

• The preferred model when considering nRMSE error as the most critical quality criterion is the RF model. This model shows an improvement of 10.562% over the naive method and uses SET1 as input data.
• The preferred model when considering nMAE error as the most critical quality criterion is the KNN model. This model shows an improvement of 1.439% over the naive method and uses SET4 as input data.
• The preferred model when both nRMSE and nMAE errors are considered as the most critical quality criteria is the XGBOOST model. The improvements over the naive method are 10.018% and 0.211%, respectively, using SET3 as input data.

Suggestions for possible changes in input data to further reduce the size of forecast errors are the following: given the strong correlation between the forecasted process and meteorological conditions, it can be assumed that meteorological forecasts might be useful in the forecasting process; also, if access to meteorological forecasts (sunlight values and temperature values every 15 min for the energy generation forecast period) is obtained, these pieces of information could serve as additional input data to improve forecast quality. Additionally, the utilization of sunlight forecasts also might considerably enhance the quality of predictions.

The authors also plan to perform quality tests on transformer-based models in the future. If access to multiple data sources from photovoltaic systems becomes available, the authors intend to analyze the effectiveness of the federated learning method [30].