An Application of Ensemble Spatiotemporal Data Mining Techniques for Rainfall Forecasting ^†

Abstract

The study proposes an ensemble spatiotemporal methodology for short-term rainfall forecasting using several data mining techniques. Initially, Spatial Kriging and CNN methods were employed to generate two spatial predictor variables. The three days prior values of these two predictors and of other selected weather-related variables were fed into six cost-sensitive classification models, SVM, Naïve Bayes, MLP, LSTM, Logistic Regression, and Random Forest, to forecast rainfall occurrence. The outperformed models, SVM, Logistic Regression, Random Forest, and LSTM, were extracted to apply Synthetic Minority Oversampling Technique to further address the class imbalance problem. The Random Forest method showed the highest test accuracy of 0.87 and the highest precision, recall and an F1-score of 0.88.

1. Introduction

Rainfall is identified as one of the most chaotic and dynamic phenomena that varies spatiotemporally [1]. Heavy and extreme rainfall creates a serious threat to human lives and properties through severe flooding. Therefore, an accurate rainfall nowcasting is of great significance in preventing devastating consequences.

The data mining techniques optimally capture the hidden spatiotemporal patterns among largely available weather-related data [2]. Many researchers achieved high prediction accuracy in rainfall classification using techniques such as Random Forest, Artificial Neural Network (ANN), K—Nearest Neighbour (KNN) and Support Vector Machine (SVM). As a recent trend, deep learning methods such as Convolutional Neural Network (CNN) and Long-Short-Term-Memory (LSTM) are employed to explore the meteorological big data due to its promising technical advantages and performance [3]. The study performed in [4] used K-means clustering to predict rainfall states. The identified clusters were used as predictands for training the Classification and Regression Tree (CART) model with five climate input variables and obtained a satisfactory value of goodness-of-fit. Another study performed CART and C4.5 models with thirteen input variables to predict the chance of rain and gained average accuracies of 99.2% (CART) and 99.3% (C4.5) [5]. Moreover, [6] modeled weekly rainfall with weather variables using ANN and produced higher prediction accuracy than multiple linear regression model. The summer precipitation patterns over eastern China were modeled using multinomial logistic regression (MLR) by [7] and gained a prediction accuracy range of (60–70%). Authors of [8] compared several machine learning models in classifying month of a year as dry or wet. The rainfall classification carried out by [9] concluded that Decision Trees and Random Forests could perform well even with a low proportion of training data. A similar study conducted by [10] extracted the Adaboost algorithm, which produced F1-score of 0.9726. Authors of [11] carried out rainfall classification addressing the class imbalance through over and under-sampling techniques, and the results indicated varying performance with different inputs generated by resampling techniques.

Our study employed several cost-sensitive machine learning models, including Penalized SVM and Complement Naïve Bayes followed by a resampling technique to address the natural rarity of extreme rainfall events. Prior to that two spatial input variables were generated by modeling satellite data with deep learning method (CNN) and rainfall at nearby rain gauging stations with Spatial Kriging. Therefore, the study focused on three solutions proposed by the literature [12] for imbalance learning. Moreover, we have performed a comparative study between machine learning and deep learning models suggested by many researchers [11,12].

The next section describes the materials and methods used during the study while Section 3 and Section 4 line up the results and discussion, and conclusions, respectively.

2. Materials and Methods

2.1. Description of Data

The weather data was obtained from the Meteorology Department of Sri Lanka based on the Kalu River basin over the period from 2015 to 2019. It includes daily data on 28 variables including rainfall at target rain gauging station (Rathnapura), rainfall of six nearby gauging stations, relative humidity, mean sea level pressure, wind speed, temperature, sunshine hours, evaporation, and Southern Oscillation Index. Additionally, the study collected daily satellite images (with a size of 500 × 512 pixels) covering the river basin from China Meteorological Administration National Satellite Meteorological Center for the same time period.

2.2. Methods

The main objective of the study is to forecast rainfall occurrences from highly imbalanced spatiotemporal time series data by using machine learning and deep learning methods. Initially, the rainfall classes needed to be identified. Therefore, with the cutoff levels established by Meteorology Department of Sri Lanka and a comparison of flood occurrence with respect to rainfall at Rathnapura, the rainfall values were categorized into three classes, ‘No rainfall’, ‘Normal rainfall’, and ‘Extreme rainfall’. Following the norm of the Department of Meteorology, Sri Lanka, the cutoff level for extreme events was set at 110 mm of rainfall. Then, the methodology illustrated in Figure 1 was carried out.

The Spatial Kriging was applied to predict current day (t) rainfall at Rathnapura using current day rainfall values of six nearby stations to incorporate spatial correlation between nearby stations and target station to final model. The previous day (t − 1) satellite image was modeled to predict the current day (t) rainfall class through CNN model. Then, the predicted rainfall class of the day t with the predicted rainfall value of the same day were brought as predictor variables to the final dataset along with other selected variables. The values of these predictor variables of last three consecutive days (t, t − 1, t − 2) were fed into six classifiers from different model families (Linear Classifier, Ensemble, and deep and sequential learning) for forecasting next day (t + 1) actual rainfall class of target variable or the response variable.

2.2.1. Spatial Kriging

Let the rainfall value in Rathnapura on a particular day $j$ be $y_j$. In Spatial Kriging estimates of $y_j$, $\widehat{{\mathit{y}}_{\mathit{j}}}$ is modeled through the rainfall values of $m$ neighboring sample locations $X_i$, i.e., $Z(X_i)$. It gives an optimal linear combination of $Z(X_i)$ with weights $w_i$, which are taken according to covariance values [13].

$$\widehat{{\mathit{y}}_{\mathit{j}}}=k+{\displaystyle \sum }_{i=1}^n{w}_{i}Z(X_i)=k+w^T\mathit{Z}$$

(1)

2.2.2. Multi-Layer Perceptron (MLP)

MLP is a feed forward neural network that consists of three types of layers, the input layer, hidden layer (s), and output layer [14]. Let us consider a MLP model (see Figure 2) with one hidden layer.

Here, the $X_j$ is the input to the $j^{th}$ neuron of the input layer, the ${w_{ij}}^{\left(1\right)}$ is the weight of the link connecting $j^{th}$ neuron of the input layer to $i^{th}$ neuron of the hidden layer, $b_i$ is the bias associated with the $i^{th}$ neuron of the hidden layer, and ${\varnothing }^{\left(1\right)}$ is the activation function associated with the hidden layer. Then, the net output from $i^{th}$ neuron of the hidden layer is given by $h_i$ (see Equation (2)). ${w_{ij}}^{\left(2\right)}$ is the weight of the link connecting $i^{th}$ neuron of the hidden layer to $j^{th}$ neuron of the output layer, $b_j$ is the bias associated with the $j^{th}$ neuron of the output layer, and ${\varnothing }^{\left(2\right)}$ is the activation function associated with the output layer. In this case, $j=1$. The output of that $j^{th}$ neuron of the output layer (or, in our case, the final rainfall prediction at Rathnapura by MLP) will be $\widehat{y_j}$ [15]. Since the network is fully connected, each unit has its own bias, and there is a weight for every pair of units in two consecutive layers. Then, the MLP network computations can be written as:

$$h_i={\varnothing }^{\left(1\right)}\left({{\displaystyle \sum }}_i{w}_{ij}^{\left(1\right)}{X}_{j }+b_i\right) \mathrm{Here}, i=1, 2, 3, 4$$

(2)

$${\widehat{\mathit{y}}}_{\mathit{j}}={\varnothing }^{\left(2\right)}\left({{\displaystyle \sum }}_i{w}_{ij}^{\left(2\right)}{h}_i+b_j\right) \mathrm{Here}, i=1, 2, 3, 4$$

(3)

2.2.3. Long Short-Term Memory (LSTM)

LSTM has four neural network layers interacting in a very special way. Its memory cell consists of a forget gate, input gate, and output gate [16].

As shown in Figure 3, the output of the last moment and current input value are fed into the forget gate to obtain the following output at the forget gate.

$$f_t=\sigma \left(w_f .\left[h_{t-1}, x_t\right]+b_f\right)$$

(4)

where $f_t\in \left(0, 1\right)$, $w_f$—weight at forget gate, $b_f$—bias at forget gate, $x_t-$ current input value, $h_{t-1}$—output at previous moment. Then, the same previous output and current input value are inputted to the input gate, and the output value and candidate cell state at the input gate are calculated as below.

$$i_t=\sigma \left(w_i .\left[h_{t-1}, x_t\right]+b_i\right)$$

(5)

$${\tilde{C}}_t=tanh\left(w_c .\left[h_{t-1}, x_t\right]+b_c\right)$$

(6)

where $i_t\in \left(0, 1\right)$, $w_i$—weight at input gate, $b_i$—bias at input gate, $w_c$—weight at candidate input gate, $b_c$—bias at candidate input gate. Update the current cell state using the following formulae.

$$C_t=f_t\ast C_{t-1}+i_t\ast {\tilde{C}}_t$$

(7)

The $h_{t-1}$ and $x_t$ are then fed into output gate at time $t$ and obtain output $O_t$ at output gate as follows. Here, $w_o$—weight at output gate and $b_o$—bias at input gate.

$$O_t=\sigma \left(w_o .\left[h_{t-1}, x_t\right]+b_o\right)$$

(8)

Finally, the output of the LSTM was obtained using the current cell state and output at the output gate using the following formulae.

$$h_t=O_t\ast tanh\left(C_t\right)$$

(9)

2.2.4. Convolutional Neural Network (CNN)

CNN is very popular for image processing and computer vision. It consists of three layers as seen in Figure 4. Convolution layer performs linear convolution operation, and the features of the data are extracted. Since the feature dimensions are very high, a pooling layer is added after the convolution layer. To make a final forecast, a fully connected (FC) layer (or dense layer) is added, and inputs to this layer are the flattened features resulted from convolutional and pooling layers [18,19].

2.2.5. Random Forest (RF)

RF is known as a supervised ensemble learning method. The method constructs a multitude of decision trees with controlled variation at the training phase. Then, using bagging, each tree in the ensemble is constructed (with sample with replacement) from training data. In the classification problem, each tree in ensemble is a base classifier to identify the class of the unlabeled observation. Through voting of each classifier for their predicted classes, the final class is obtained computing the majority votes [20,21].

2.2.6. Support Vector Machine (SVM)

SVM classifier finds a hyperplane to segregate the nodes for classification. The following optimization problem is solved when deriving the optimal hyperplane which separates two classes.

$$\underset{w, b,{\epsilon }_j}{\mathrm{Minimize}}\frac{1}{2}{w}^{T}w+C{\displaystyle \sum }_j{\epsilon }_j$$

(10)

$$\mathrm{Subject} \mathrm{to} y_j\left(w^T\mathsf{\Phi }\left(X_j\right)+b\right) \ge 1-{\epsilon }_j$$

(11)

$${\epsilon }_j \ge 0$$

(12)

where $y_j$ is the class label, $w$ is the weights vector, $X_j$ is the input feature vector, $\mathsf{\Phi }$ is the transformation function, ${\epsilon }_j$ is the degree of misclassification corresponding to $X_j$, $C$ is the regularization parameter and $b$ is the bias.

The optimal hyperplane (a maximum marginal hyperplane (MMH)) is learnt by training the samples using several kernel functions such as linear, radial basis function (rbf) and polynomial (poly) [22]. The Penalized SVM (PSVM or Cost-sensitive SVM) is a modification of SVM that weighs the margin proportional to the class importance which can be applied to an imbalanced dataset [23,24].

2.2.7. Naïve Bayes (NB)

The Naïve Bayes Algorithm is based on the Bayes Theorem of probability.

$$Pr\left(C|X\right)=\frac{Pr\left(X|C\right)\mathit{Pr}\left(C\right) }{Pr\left(X\right) }$$

(13)

where, $Pr\left(C|X\right)$: The posterior probability of the class of interest given predictor, $\mathit{Pr}\left(C\right)$: The prior probability of the class of interest, $Pr\left(X|C\right)$: The probability of predictor given class C, and $Pr\left(X\right)$: The prior probability of predictor.

NB method calculates the probability of an observation belonging to a certain class. The Complement Naïve Bayes (CNB) method computes the probability of the observation belonging to all the classes. Thus, CNB is more suitable in dealing with imbalanced datasets [25].

2.2.8. Multinomial Logistic Regression (MLR)

Multinomial Logistic Regression is the generalization of Logistic regression which allows more than two categories of output variable. It also obtains maximum likelihood estimation to evaluate the probability of categorical membership [26]. The formula of MLR is as follows:

$$\log \left[\frac{P{r}_i\left(Y\right)}{P{r}_j\left(Y\right)}\right]={\alpha }_i+{\beta }_i^{T}X$$

(14)

where $P{r}_i\left(Y\right)$ is the probability of $Y$ in $i^{th}$ class, $P{r}_j\left(Y\right)$ is the probability of $Y$ in $j^{th}$ class, ${\alpha }_i$ is the intercept, ${\beta }_i^T$ is the vector of covariates for the $i^{th}$ class of the output variable $Y$, and $X$ is the input feature vector [27].

2.3. Imbalanced Learning

The data mining techniques will produce biased classification if the dataset is imbalanced [11]. Moreover, it can lead to a problem in ignoring the minority class entirely in the case where the predictions on the minority class are most important. This is a major issue found in rainfall forecasting. Certain methodologies can deal with the class imbalance of the data.

2.3.1. Cost—Sensitive Learning

This tactic uses penalized learning algorithms which give higher misclassification costs (or weights) for instances of the minority class and lower misclassification costs for the majority class [12,24].

2.3.2. Resampling Techniques

The resampling techniques are applied to obtain more balanced datasets. In this study, Synthetic Minority Oversampling Technique (SMOTE) which synthesizes new samples from the minority class was applied.

2.4. Model Evaluation

The metrics, Accuracy, Precision, Recall and F1-Score were used to evaluate the classification models, and the Spatial Kriging model results were evaluated using Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and R² value.

Firstly, the cost-sensitive approach was followed in rainfall class prediction. Through the model evaluation results, the best set of models were chosen to apply the resampling technique. The final evaluation based on resampling was taken into consideration in selecting the best model for rainfall classification. Before applying the classification models, all the input variables were normalized. The machine learning and deep learning algorithms were run in Python.

3. Results and Discussion

As mentioned previously, initially the Spatial Kriging method was applied to find the daily rainfall prediction at Rathnapura gauging station.

The results shown in

Table 1. Performance of Spatial Kriging method.

Metric	RMSE	MAE	R2
Error	16.2	6.64	30.3%

indicate that the Spatial Kriging model fitted using the rainfall values of nearby stations cannot be solely used to explain the variation of the rainfall at target station, yet, they have an influence on the target station’s rainfall.

Then, the previous day’s (t − 1) satellite image was modeled with the current day’s (t) rainfall occurrence in Rathnapura using the CNN model. The best model parameters obtained via 60 trials of training the models CNN, MLP and LSTM are presented in

Table 2. Parameter specification of CNN, MLP and LSTM models.

Parameters	CNN	MLP	LSTM
Activation function	Softmax	Softmax	Softmax
Epochs	50	50	50
Number of hidden layers	7	2	1
Number of neurons in final dense function	3	3	3
Batch size	72	72	72
Kernel size	(3,3)	-	-
Learning rate	0.01	0.01	0.01
Optimizer	Adam	Adam	Adam
Loss Function	Categorical Cross Entropy	Categorical Cross Entropy	Categorical Cross Entropy

.

The CNN model showed 64.9% of Accuracy and Recall with 59.7% of Precision and 52.8% of F1-Score. The results also suggest the same conclusion produced by the Spatial Kriging method.

However, applying Spatial Kriging reduced the dimensions (number of input variables) of the final model. This method along with satellite images analysis are set to incorporate the spatial variation of the rainfall data to the final model.

The predictions obtained from the above two spatial models were incorporated as new predictor variables to the final dataset. Then, there were 23 predictor variables. The values of the past three days (t, t − 1, t − 2 on (2)) spatial correlation between nearby stations and target station to final model. t − 2) of each predictor variables were modeled with the next day (t + 1) actual rainfall class since through a preliminary data analysis we could identify that the past three days rainfall values have much impact on the next day (t + 1) rainfall occurrence.

To address the class imbalance, cost-sensitive models were applied. The entire data set was split as 80% for training and 20% for testing. For the training set, Repeated Stratified 5—Fold Cross Validation (which repeats the cross-validation procedure multiple times) was applied to further address the class imbalance problem.

Table 3. The performance of cost-sensitive models in training sets.

Metric	Cost-Sensitive Model
	PSVM (Kernel = rbf)	PSVM (Kernel = Poly)	CNB	MLR	RF	MLP	LSTM
Accuracy	0.75	0.73	0.71	0.74	0.73	0.73	0.73
Recall	0.75	0.74	0.72	0.74	0.74	0.74	0.74
Precision	0.75	0.73	0.71	0.74	0.73	0.73	0.73
F1-Score	0.75	0.73	0.71	0.73	0.73	0.73	0.73

and

Table 4. The performance of cost-sensitive models in test set.

Metric	Cost-Sensitive Model
	PSVM (Kernel = rbf)	PSVM (Kernel = poly)	CNB	MLR	RF	MLP	LSTM
Accuracy	0.71	0.77	0.68	0.81	0.76	0.70	0.71
Precision	0.72	0.75	0.75	0.79	0.75	0.72	0.75
Recall	0.71	0.77	0.68	0.81	0.76	0.70	0.71
F1-Score	0.72	0.76	0.71	0.78	0.75	0.71	0.73

show model performance.

The training and testing performances indicate that cost-sensitive SVM, Random Forest, Multinomial Logistic Regression, and LSTM models have performed better in terms of metrics, especially precision, recall, and F1-score, which are more suitable in evaluating class-imbalanced problems [28]. The selected models depict more than 70% Accuracy, Precision, Recall, and F1-Score.

Then, for the selected models, the SMOTE resampling technique was applied. The performance was evaluated after refitting the balanced dataset using the selected best set of models. The following

Table 5. The performance of cost-sensitive resampled models in training set.

Metric	Cost-Sensitive Resampled Model
	PSVM (Kernel = rbf)	PSVM (Kernel = Poly)	MLR	RF	LSTM
Accuracy	0.82	0.83	0.80	0.87	0.77
Precision	0.82	0.82	0.79	0.87	0.78
Recall	0.82	0.83	0.80	0.87	0.79
F1-Score	0.82	0.83	0.80	0.87	0.77

and

Table 6. The performance of cost-sensitive resampled models in test set.

Metric	Cost-Sensitive Resampled Model
	PSVM (Kernel = rbf)	PSVM (Kernel = Poly)	MLR	RF	LSTM
Accuracy	0.82	0.82	0.79	0.87	0.78
Precision	0.82	0.83	0.79	0.88	0.79
Recall	0.82	0.83	0.80	0.88	0.79
F1-Score	0.82	0.83	0.79	0.88	0.78

illustrate the final performance results.

It can be observed that after two operations, the performance of all selected models has improved. Out of them, the Random Forest method gives the best and consistent performance in both the training set and in the final evaluation (in test set) of the selected models (nearly 88% of Accuracy, Precision, Recall, and F1-Score).

Overall, the study results indicate the importance of incorporating spatial variation of the rainfall data in predicting future events and highlight the effectiveness of step-wise imbalance learning to obtain consistent and more accurate predictions which could not be attained in some previous studies.

4. Conclusions

In this study, we proposed a novel ensemble spatiotemporal data mining approach to forecast rainfall occurrence at the Rathnapura gauging station. Spatial Kriging and a Deep Learning model (CNN) were employed to capture the spatial variation over the selected grid. The temporal variation of the rainfall data was brought to the model by modeling with the three past consecutive days’ values of the variables. Five cost-sensitive models were further improved to address the imbalanced problem found in rainfall classes through a resampling technique. The final performance summary emphasizes the outperformance of the cost-sensitive resampled Random Forest method (nearly 88% of accuracy, precision, recall, and F1-score) in forecasting future rainfall occurrences.

During the study, we found the complexity of working with high number of predictor variables. Therefore, our future studies are expected to enhance further by focusing on feature selection and application of dimension reduction prior to the model application. Collecting data for an extended period (e.g., 30 years) and selecting novel approaches will also be taken into consideration when dealing with highly imbalanced rainfall data.