A Comparison of the Statistical Downscaling and Long-Short-Term-Memory Artificial Neural Network Models for Long-Term Temperature and Precipitations Forecasting

Abstract

General circulation models (GCMs) run at regional resolution or at a continental scale. Therefore, these results cannot be used directly for local temperatures and precipitation prediction. Downscaling techniques are required to calibrate GCMs. Statistical downscaling models (SDSM) are the most widely used for bias correction of GCMs. However, few studies have compared SDSM with multi-layer perceptron artificial neural networks and in most of these studies, results indicate that SDSM outperform other approaches. This paper investigates an alternative architecture of neural networks, namely the long-short-term memory (LSTM), to forecast two critical climate variables, namely temperature and precipitation, with an application to five climate gauging stations in the Lake Chad Basin. Lake Chad is a data scarce area which has been impacted by severe drought, where water resources have been influenced by climate change and recent agricultural expansion. SDSM was used as the benchmark in this paper for temperature and precipitation downscaling for monthly time–scales weather prediction, using grid resolution GCM output at a 5 degrees latitude × 5 degrees longitude global grid. Three performance indicators were used in this study, namely the root mean square error (RMSE), to measure the sensitivity of the model to outliers, the mean absolute percentage error (MAPE), to estimate the overall performance of the predictions, as well as the Nash Sutcliffe Efficiency (NSE), which is a standard measure used in the field of climate forecasting. Results on the validation set for SDSM and test set for LSTM indicated that LSTM produced better accuracy on average compared to SDSM. For precipitation forecasting, the average RMSE and MAPE for LSTM were 33.21 mm and 24.82% respectively, while the average RMSE and MAPE for SDSM were 53.32 mm and 34.62% respectively. In terms of three year ahead minimum temperature forecasts, LSTM presents an average RMSE of 4.96 degree celsius and an average MAPE of 27.16%, while SDSM presents an average RMSE of 8.58 degree celsius and an average MAPE of 12.83%. For maximum temperatures forecast, LSTM presents an average RMSE of 4.27 degree celsius and an average MAPE of 11.09 percent, while SDSM presents an average RMSE of 9.93 degree celsius and an average RMSE of 12.07%. Given the results, LSTM may be a suitable alternative approach to downscale global climate simulation models’ output, to improve water management and long-term temperature and precipitations forecasting at local level.

1. Introduction

The climate system is highly complex, dynamic and multi-dimensional, with no single steady state. Models that attempt to predict the expected state of the system are therefore not intended to be an exact representation of the system, or able to reproduce all aspects of the system perfectly. However, partial representations and predictions are still useful in understanding certain phenomena, as well as in financial and infrastructure planning and adapting to climatic change, which has long been recognised as being crucial in society and the environment [1]. Where we may not be able to perfectly predict the climate system, we can attempt to improve our prediction methods for small parts of the system to better support decision making, especially for regional or local-level analyses [2]. Considerable efforts have been devoted to improving long-term weather forecasting. General circulation models (GCMs) also known as global climate models are established techniques for these types of assessments [3,4,5]. GCMs use the Navier–Stokes equations of thermodynamics, to understand climate patterns, given various energy sources such as radiation and latent heat [6,7]. GCMs produce large scale forecasts which are generally too coarse in resolution and do not consider processes on a small scale for localized use. To make GCM output more useful on a regional or local level, downscaling methods of GCM output have been developed that is either dynamic and based on regional climate models [8], or statistical, or a combination of both [9]. In this article, the focus is on statistical methods of downscaling.

Wilby [10] developed the Statistical Downscaling Model (SDSM), which has since been widely applied to temperature and precipitations forecasting [11,12,13]. Statistical downscaling is the process of using GCM atmospheric output, to estimate precipitations, maximum temperatures as well as minimum temperatures at local level [14]. Different techniques for statistical downscaling have since been developed and are described in several textbooks and review publications [9,15,16,17]. In recent years, statistical downscaling and bias correction have become standard tools in climate impact studies. The growing field of machine learning has also received increasing attention in geoscientific studies [18,19]. Harphan and Wilby [20] compared SDSM to multi-layer perceptron (MLP) neural network for precipitations forecasting, with an application to precipitation gauging stations in England. Contrasting results were found between stations. A number of other studies comparing both the MLP as well as the recurrent neural network (RNN) with SDSM were recorded [21,22,23]. In all these cases, SDSM outperformed both MLP and RNN in downscaling precipitation, minimum and maximum temperatures forecasting.

Long-Short-Term-Memory (LSTM) was proposed by Hochreiter and Schmidhuber [24] in 1997 and has been shown superior in learning long-term dependencies between inputs and outputs as compared to MLP and RNN, given its specific architecture, which consists of a set of recurrently connected subnets, known as memory blocks [25]. Bengio [26] shows that it is difficult for RNN to remember a sequence of more than 10 lags. Grave [25] shows in contrast that each LSTM block contains one or more self-connected memory cells and three multiplicative units, namely the input, output and forget gates, which allow longer memory. However, very few cases are reported in literature where LSTM is applied to hydrology and climate studies [27,28]. Given the superior performance of LSTM in learning long-term dependencies compared to MLP and RNN, the aim of this article is to compare LSTM prediction performances for long-term precipitation, minimum and maximum temperature forecasting, against SDSM, to establish whether LSTM can outperform SDSM, in contrast to the cases of MLP and RNN. In addition, given the limited applications of LSTM to hydrological and climate-related studies in literature, this study contributes to literature by establishing the relevance of LSTM within the given context, as an alternative to the widely used SDSM. Furthermore, by using the Lake Chad Basin as a case study, our approach specifically test SDSM and LSTM performances for climate stations with significant differences in Mean Annual Precipitation (MAP: 510–1160 mm/yr) and average monthly temperatures (24.5 to 41 °C) and investigate the drivers of these differences. The catchment has a high spatial variability in precipitation, ranging from 150 mm/yr in the north to over 1500 mm/yr in the South [29]. Likewise, monthly average temperature ranges from 40.8 °C in the north to 24.5 °C in the south [30].

The main objective of this study is therefore to compare the performances of SDSM and LSTM in downscaling monthly precipitation, minimum and maximum temperatures for selected gauging stations in the Lake Chad Basin. In order to compare SDSM and LSTM performances, the root mean square error (RMSE) as well as the mean absolute percentage error (MAPE) and the Nash Sutcliffe Efficiency (NSE) are used as performance indicators. The interest of using all three resides in the fact that the RMSE measures the ability of the model to avoid large errors between the prediction and the actual values, the MAPE provides an idea on the overall error percentage in the prediction, and the NSE is a standard measure used in the field of climate forecasting and hydrology for comparison [31]. The presented approach provides an enhanced downscaled GCM product, as well as the modelling framework needed to develop better local predictions of precipitation and temperature from GCMs, considering data and computing limitations.

2. Study Area and Data

2.1. Study Area

The Lake Chad Basin is a trans-boundary basin, situated between Cameroon, Chad, Niger and Nigeria. It is one of the largest interior basins in the world, with an estimated catchment area of 2,500,000 km${}^2$. The lake itself supports 37 million people with water resources which are needed for agriculture, fishing and domestic use [32]. It contains two rivers upstream, namely the Logone and Chari, and a lake downstream called Lake Chad. On the northern catchment, the Kumadugu-Yobe river is considered as a marginal water contributor in the basin, with only one percent contribution, due to its divertion in Nigeria for national food production. Water for agriculture and domestic water use is typically taken directly from the lake, its main feeding tributaries, but also from a highly productive unconfined quaternary aquifer [29]. The southern catchment provides food and income for riparians living in cities like Ndjamena, through irrigated agriculture and fishing activities. Food security in this region requires a sustainable management of available water resources. This is especially important, considering the fact that the Lake Chad is in a water scarce region.

In the last half century, the lake’s surface area has decreased from 24,000 km${}^2$ in the 1950s to 1700 km${}^2$ currently. Inconsistent trends between precipitation and the lake’s water level remain, particularly during dry periods [33], but significant trends have been associated between the lake’s water level recovery and the occurrence of wet periods from 2012–2014 [34]. The increasing use of irrigation and abstraction from the lake itself are additional major threats contributing to the lake’s decline and vulnerability to climate change [35]. Being able to forecast temperature and precipitation, two critical climate variables are required for improving water management, to forecast oncoming dry or wet periods but also reduce the over allocation of water.

For our study, only five out of nine gauging stations are considered, given data availability and quality. The five stations considered in the study are presented in Figure 1 and enumerated from upstream to downstream as follows: Sarh, Doba, Lai, Bongor and Ndjamena. The MAP for the locations Sarh, Doba, Lai, Bongor and Ndjamena is 1160 mm/yr, 1381 mm/yr, 1100 mm/yr, 1018 mm/yr and 510 mm/yr. The average monthly temperature for these locations are 30.4 °C, 39.0 °C, 33.4 °C, 39.4 °C, and 40.8 °C for Sarh, Doba, Lai, Bongor and Ndjamena.

2.2. Data

The data used in this paper are divided in two sets, namely: (1) The large-scale atmospheric data used as independent variables [36]; as well as (2) the precipitations, maximum and minimum temperatures, observed at-site, collected by the Lake Chad Basin Commission (LCBC) [30]. The latter are used as dependent variables [30] in the SDSM and LSTM models.The atmospheric data are 57 years (1950–2007) of monthly atmospheric parameters collected from the national centre for environmental prediction (NCEP), for each gauging station, using longitude and latitude coordinates for Sarh (9.14 N 18.36 E), Doba (8.67 N 16.85 E), Lai (9.39 N 16.3 E), Bongor (10.27 N 15.37 E) and Ndjamena (12.13 N 15.05 E). The data from the LCBC are 57 years (1950–2007) of monthly data recorded at each gauging station. For the SDSM implementation, the calibration set consists of 70.2 percent of the original dataset, and the validation set 29.8 percent. For the LSTM implementations, the training, validation and test sets consist of $70.20$, $24.56$ and $5.26$ percent of the data, respectively (see

Table 1. Data subsets for the purpose of comparison between SDSM and LSTM implementation in the Lake Chad Basin. The validation set of the SDSM implementation correspond to the test set of the LSTM subset.

Methods	SDSM		LSTM
Subset Names	Calibration	Validation	Training	Validation	Test
Years	1950–2004	2005–2007	1950–1990	1991–2004	2005–2007
Data Allocated (%)	$94.74$	$5.26$	$70.20$	$24.56$	$5.26$

). For the purpose of clearer comparison between SDSM and LSTM, the SDSM was re-implemented using only the last three years of the SDSM validation, to correspond with the test set of the LSTM implementation (see Table 1).

For the chosen training, validation and test subsets, there is a consistent statistically significant difference in variance between the calibration and validation sets in the five gauging stations. This significant difference in variance may be explained by the fact that the validation set coincides with a set of extreme values, caused by the drought that happened between 2006 and 2007.

3. Methods

3.1. Statistical Downscaling Model

SDSM developed by Wilby [10] is a hybrid of multiple linear regression (MLR) and the stochastic weather generator (SWG). SDSM is presented in the literature as the easiest to implement, with low computational requirement and yet satisfactory accuracy. MLR designs the econometric relationship between the NCEP data and the at-site predictants, while SWG on the other hand uses the MLR parameters to generate up to 20 climate scenarios, over a period of 3 years. The SDSM approach is summarised in four steps: (1) screening, (2) calibration, (3) validation, (4) climate scenarios simulation.

The screening step consists of selecting the atmospheric data presenting the highest correlation with the variables to predict. A correlation matrix is used to filter the 31 independent variables (atmospheric data), such that only the predictors with correlation greater than 30 percent with at-site dependent variables and presenting very little paired correlation between each other are used for downscaling purposes. Predictors selected are presented in

Table 2. Definition of variable names, their units and corresponding models. °N denotes North direction, mm denotes Millimeter, and * denotes dimensionless Z-score. (a) denotes monthly precipitation model, (b) denotes monthly minimum temperatures and (c) denotes monthly maximum temperatures model.

Variable	Definition	Units	Model
Min temp	Minimum temperatures	degree celsius	(c)
Max_temp	Maximum temperatures	degree celsius	(b)
dswr	Direct shortwave radiation	*	(a,b,c)
lftx	Surface lifted index	*	(a)
p_u	Zonal velocity component near the surface	*	(a)
p_th	Wind direction near the surface	°N	(a)
p5_f	Geostrophic airflow velocity at 500 hPa	*	(a)
p5_u	Zonal velocity component at 500 hPa	*	(a)
p5_z	Vorticity at 500 hPa	*	(a)
p5th	Wind direction at 500 hPa	°N	(a)
p8_u	Zonal velocity component at 850 hPa	*	(a)
p8th	Wind direction at 850 hPa	°N	(a)
pr_wtr	Precipitable water	*	(a)
ncepprec	Large scale precipitations	mm	(a)
r500	Large scale precipitations	*	(a)
r850	Relative humidity at 500 hpa height	*	(a)
rhum	Relative humidity at 850 hpa height	*	(a)
shum	Near surface relative humidity	*	(a)
mslp	Mean sea level pressure	*	(b,c)
p_z	Vorticity near the surface	*	(b,c)
pottmp	Potential temperature	*	(b,c)
p500	500 hpa geopotential height	*	(b,c)

along with the corresponding models where they are used. The precipitation forecasting model is denoted by (a), the minimum temperature model is represented by (b) and the maximum temperature forecast model is denoted by (c). The selected variables are used in SDSM4.2 [36] to predict precipitations, minimum and maximum temperatures three years ahead.

The calibration step consists of finding the suitable parameters that explain the causal relationship between the selected predictors (see Table 2) and each predictant, namely precipitations, minimum temperatures and maximum temperatures. The calibration principle in the SDSM software (SDSM4.2) consists of splitting the dataset into calibration and validation, and then splitting the calibration set into k subsets, $k\in \mathbb{N}$, such that $k-1$ subsets are used for calibration and the $k\mathrm{th}$ subset for internal validation. With such configuration, k calibrated models are built, and the average parameters are considered for prediction purposes. SDSM4.2 allows up to $k=20$ folds. This study therefore performs 20 calibrations and internal validation.

The calibration robustness is given by the average coefficient of determination ($R^2$) from the 20 calibrated models; the model accuracy is given by the average standard error (SE) from the 20 internal validation subsets. The residual autocorrelation test is performed on each of the 20 calibrated models to check that the residuals are white noise and therefore there is absence of lag one residuals autocorrelation, in the models used to predict precipitations, minimum temperatures as well as maximum temperatures. The Durbin-Watson test is used to check the residuals for autocorrelation [37]. The parameters stability test is also performed to check whether the parameters obtained do not vary significantly with sample change. The Chow tests is performed for the parameter stability check [38]. Once the tests are successfully performed on the calibration set, the average regression parameters explaining the causal relationship between the atmospheric data and each dependent variable, namely precipitations, minimum and maximum temperatures, are applied to the validation set. The aim is to check the capacity of the model to predict unseen data with accuracy. This is investigated by comparing the observed values with the model output. The validation tests include the Levene test for variance comparison and the Student paired t-test test for mean comparison. The simulation scenarios use the validated model and generate up to 20 scenarios of future precipitations or temperatures.

The above mentioned procedure is performed to show that SDSM results were obtained through a rigourous process, and can therefore be compared with results from LSTM without bias.

3.2. Long-Short-Term Memory Network

In this paper, three implementations of LSTM to forecast precipitations, minimum temperatures and maximum temperatures, respectively, are presented. The following sections provide details on (1) the architecture, (2) the feedforward propagation, (3) the loss function and (4) the backpropagation and weights update used in the three different implementations.

3.2.1. LSTM Architecture and Feedforward

Figure 2 presents a classical hidden layer node with an activation function (A) and a single cell LSTM memory block with forget gate, input gate and output gate. In the figure, $c_t$ denotes the cell state that contains all previous values of atmospheric data, such that:

$$c_{t-1}=\left[\begin{array}{ccccc}{x}_{11}& x_{12}& x_{13}& \dots & x_{1t-1}\\ x_{21}& x_{22}& x_{23}& \dots & x_{2t-1}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ x_{i1}& x_{i2}& x_{i3}& \dots & x_{it-1}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ x_{I1}& x_{I2}& x_{I3}& \dots & x_{It-1}\end{array}\right],$$

where $x_{it-1}$ is the general term of the ith predictor at time $t-1$, with $i\in I$ and $t-1\le T$. It is worthwhile to note that I is the set of predictors and T is the length of time of a specific predictor. In this case study, the number of predictors is $16,6,\mathrm{and}6$ for precipitations, maximum and minimum temperatures respectively (see number of predictors for model (a), (b), (c) in Table 2).

An effective use of the information contained in the cell state for actual prediction at time t is performed through three types of decisions which are taken at three gates in the LSTM architecture, namely the forget gate, the input gate and the output gate.

Forget gate: This gate is used to decide on the atmospheric data points to throw away from the cell state $c_{t-1}$ in the estimation process of the target value ${\widehat{y}}_t$. A sigmoid function ($\sigma$) is used for the purpose, with

$$f_t=\sigma (W_f{x}_t+U_f{h}_{t-1}+b_f).$$

(1)

In Equation (1), $f_t$ is a vector with values in the range (0,1), $W_f$ the matrix of the learnable coefficients associated with the vector of predictor inputs $x_t$ (atmospheric predictors at time t), $U_f$ is the hidden layer vectors $h_{t-1}$, representing all matrix of predictors value up to $x_{t-1}$ (atmospheric predictors before time t) and $b_f$ is the matrix of biased coefficients associated to each hidden layer, at the forget gate f. At time step $t=1$, the hidden layer is given by a null vector of length I, and the potential cell state vector denoted by $\widetilde{c_t}=0$, given that there is no previous information in the network. For $t\in \{2,3,\dots ,T\}$, $\widetilde{c_t}=tanh(W_{\tilde{c}}{x}_t+U_{\tilde{c}}{h}_{t-1}+b_{\tilde{c}})$, such that $\tilde{c}$ is a vector with values between $(-1,1)$, $W_{\tilde{c}}$, $U_{\tilde{c}}$ and $b_{\widetilde{c_t}}$ are another set of learnable parameters.

Input gate: This gate decides on what new information from atmospheric data should be used to replace the information removed from the cell state at the forget gate. It uses both a sigmoid function to decide which values to update, and a hyperbolic tangent function ($tanh$), to create a vector of new cell state candidate values, denoted by $c_t$. The computation used at the input gate to update the cell state is given by

$$i_t=\sigma (W_i{x}_t+U_i{h}_{t-1}+b_i),$$

(2)

where $i_t$ is a vector with values in range (0,1), $W_i$, $U_i$ and $b_i$ are learnable parameters defined for the input gate.

The cell state update from $c_{t-1}$ to $c_t$ is finally given by

$$c_t=f_t.c_{t-1}+i_t.\widetilde{c_t}.$$

(3)

Output gate: The output gate controls the information flow from $c_t$ to the next hidden layer $h_t$ such that

$$o_t=\sigma (W_o{x}_t+U_o{h}_{t-1}+b_o)$$

(4)

where $o_t$ is a vector with values in the range (0,1), $W_o$, $U_o$ and $b_o$ are learnable parameters for the output gate. A new hidden state is computed for the next prediction, such that

$$h_t=tanh(c_t).o_t.$$

(5)

The single output neuron at the last time step (${\widehat{y}}_t$), which computes either the precipitation, the maximum or minimum temperatures, is given by the last layer of the LSTM, denoted the dense layer, such that

$${\widehat{y}}_t=W_d{h}_t+b_d,$$

(6)

where $W_d$ is the matrix of the learnable weights of the dense layer, $b_d$ is the biased term. The computation of ${\widehat{y}}_t$ is followed by the computation of the error function.

3.2.2. Error Function

The output of the activation function at the last layer is compared to the target solution, given in historical data. The error function at the kth iteration ($E_k$), is the average deviation obtained for each sample $(y_t,{\widehat{y}}_t)$, as given by

$$E_k=\frac{1}{T}\sum _{t=1}^T{(y_t-{\widehat{y}}_t)}^2.$$

(7)

Here, the objective is to find a linear combination of weights and inputs, that minimises the error function.

Backpropagation is used to determine the contribution of each neuron per layer to the error function. It allows weight update, in order to increase forecasting accuracy.

3.2.3. The Backpropagation in LSTM

This study uses the mini batch gradient descent, as it updates weights for every mini-batch of m training examples ($m\in T$, $m<T$), and ensures more stable convergence to the minimised error function. Mini-batch is a good trade-off optimisation algorithm for error function minimisation [40]. This study uses mini batch sizes of $m=4,5$ and 10 (see

Table 3. Architectures and hyperparameters investigated for monthly precipitation and temperatures forecasting.

Hyperparameter & Architecture	Options Explored	Best Selections
Window size	1, 2, …, 12, 24, 36	36
Hidden Nodes	5, 10, 15, 20, 25, 30, 35	15, 20
Optimizers	adam, adagrad, sgd, adadelta, Nadam	adam
LearningRates	0.1, 0.01, 0.001, 0.09, 0.2	0.01
Dropouts	0.09, 0.1, 0.2, 0.3, 0.4, 0.5	0.09, 0.2
Batch sizes	4, 5, 10, 20, 25, 30	5, 10, 15
Epochs	100, 200, 500, 1000, 2000, 2500, 3000	2500, 3000

).

In order to minimise the error function in Equation (7), partial derivatives with respect to the parameters of the weight matrices ($W_f,U_f,b_f,W_i,U_i,b_i,W_{\tilde{c}},U_{\tilde{c}},b_{\tilde{c}},W_o,U_o,b_o$) are computed. Previous weights are updated according to both the sign of the derivative, and the learning rate ($\nu$), given as input to the network, in order to speed up the convergence in the optimisation direction.

A summary of the implementation process of the LSTM methodology is presented in Algorithm 1. Python 3.5 [41], Pandas [42], Keras [43] and tensorFlow [44] libraries are used for the implementation.

Algorithm 1 Steps to implement LSTM in Python

Import Packages Import and transform time series to Dataframe Transform Dataframe to supervised series Normalise data to scale $[0;1]$ Split Data into Training, Validation and Test subsets Use the Training and Validation sets to define the LSTM model Compile and fit the model Tune the parameters until $\Delta E_k\to 0$ Use the model to perform the predictions on the test set Perform 30 simulations and store the output Compute average performances of the simulation outputs

3.2.4. LSTM Setup

In this work, we developed a basic neural network model based on the LSTM framework. The model contains 3 layers (Figure 3). LSTM layers with 576 and 216 neurons were set as the input layer, for precipitations and temperatures forecasting, respectively. A dropout rate of $0.2$ was set on the LSTM layer. A fully connected layer was then set up as the output layer which yields three time series, namely precipitations, minimum and maximum temperatures.

3.2.5. Tuning Procedure for LSTM Window Size and Hyperparameters

A proper setting of the window size is important for good model performances. In this study, monthly window sizes, of 1, 2, …, 12, 24, and 36 months, were explored (see Table 3). The window size of 36 months, that produce the best forecasting accuracy were then selected for each forecasting task. Hyperparameter optimization or tuning is the process used to find a tuple of hyperparameters that yields a model which minimize the loss function on the given data [45]. The mean-square-error was used in this study as the loss function for hyperparameter optimization [27].

In this study, after investigating five options of optimisers (see Table 3), we chose the efficient adam version of stochastic gradient descent [46]. The epoch, which is generally defined as one pass over the entire dataset in neural network, is used to separate training into distinct phases. Training for too long will lead the model to be overfitted, learning patterns that only exist in the training dataset. Whereas training for too short will lead the model to be underfitted, which means the model has not learned the relevant patterns in the training data [45]. In this study, the training epochs were first set to 200 [27] and the highest NSE were achieved after 2500, and 3000 epochs (see

Table 4. The chosen LSTM architectures for precipitation, minimum and maximum temperature forecasting in the five gauging stations after hyperparameter tuning.

	Window Size	Nodes	Epochs	Hidden Layers	Optimiser	Dropout	Batch Size	Simulations
Precipitation
Sarh	36	20	3000	1	adam	0.09	4	20
Bongor	36	15	3000	1	adam	0.2	10	20
Doba	36	20	2500	1	adam	0.09	5	20
Lai	36	15	3000	1	adam	0.2	10	20
Ndjamena	36	15	3000	1	adam	0.2	10	20
Minimum temperature
Sarh	36	15	3000	1	adam	0.2	10	20
Bongor	36	15	3000	1	adam	0.2	10	20
Doba	36	15	3000	1	adam	0.2	10	20
Lai	36	15	2500	1	adam	0.2	10	20
Ndjamena	36	15	3000	1	adam	0.2	10	20
Maximum temperature
Sarh	36	15	3000	1	adam	0.2	10	20
Bongor	36	15	3000	1	adam	0.2	10	20
Doba	36	15	2500	1	adam	0.2	5	20
Lai	36	15	2500	1	adam	0.2	10	20
Ndjamena	36	15	3000	1	adam	0.2	10	20

and Figure 4), in the five gauging stations, for precipitations and temperature forecasting. Consequently, we used the number of 2500, and 3000 epochs for the final training of the model. These number of epochs were selected after a continuous stability of the training and validation loss, over more than 1500 iterations, after an unstable learning stage that occurred between 0 to 1000 iterations, represented with an oscillating validation loss function, as observed in Doba and Bongor (see Figure 4).

The best network architecture per gauging station for precipitation is given in Table 4. They indicate heterogenous architectures per gauging stations. These architectures were obtained after trials and errors of multiple configurations. The window size of 36 months was investigated for all stations, with one hidden layer and adam optimiser. Twenty nodes were found suitable for Sarh and Doba, while fifteen nodes were found suitable for Bongor, Lai and Ndjamena. An identical number of epochs were found to produce best results for all stations, except for Doba, where fewer epochs were used. Different dropout and batch size strategies were found for each gauging station. The ensemble method was implemented for twenty simulations per stations. The LSTM architectures for minimum and maximum temperatures are also given in Table 4. Almost identical architectures were implemented for both minimum and maximum temperatures forecasting. A fewer number of epochs (2500) were considered in Lai for minimum temperature forecasting. A fewer number of epochs in both Doba and Lai (2500 epochs), for maximum temperatures forecasting, and a fewer number of batch size in Doba (5 batch size).

3.3. Models Comparison

The results from the three LSTM models for precipitation, minimum and maximum temperatures forecasting are compared with results from the SDSM using the root mean squared Error (RMSE) as well as the mean absolute percentage error (MAPE), and the Nash Sutcliffe Efficiency (NSE). The corresponding mathematical formulations are given by

$$RMSE=\sqrt{\frac{1}{N}\sum _{t=1}^N{\left(x_t-s_t\right)}^2},$$

(8)

$$MAPE(\%)=\frac{100}{N}\frac{{\sum }_{t=1}^N(|x_t-s_t|)}{x_t},$$

(9)

$$NSE=1-\frac{{\sum }_{t=1}^N{\left(x_t-s_t\right)}^2}{{\sum }_{t=1}^N{\left(x_t-\overline{x_t}\right)}^2},$$

(10)

where $x_t$ is the observed data, $s_t$ is the generated data, and N is the length of the dataset.

The model with the smallest MAPE, RMSE or the highest NSE was considered as the most accurate, and was adopted for precipitations and temperatures forecasting in the given gauging station.

4. Results

4.1. SDSM Results

The SDSM models’ performances on the training and validation precipitation subsets are presented in Figure 5. The mean RMSE, the mean MAPE, the mean NSE, as well as the mean coefficients of determination ($R^2$) for 20-folds precipitations forecasting are given for each of the five gauging stations. There is a contrast between both the high NSE and $R^2$, on the calibration set, and the low NSE and $R^2$, on the validation set, in the five gauging stations (NSE = 0.70 versus NSE = 0.51; and $R^2=88.81\%$ versus $R^2=30.61\%$). This is an indication of the weak robustness of the SDSM precipitation forecasting models. This may be explained by a number of aspects, including a consistent statistically significant difference in variance between the calibration and validation sets in the five gauging stations (see

Table 5. The statistical difference of the mean and variance estimation for the five gauging stations between the training, validation and test sets. p-values are given in brackets.

	Training vs. Validation		Validation vs. Test		Training vs. Test
	$\mathbf{\Delta }\overline{\mathit{X}}$	$\mathbf{\Delta }\mathit{var}\mathbf{\left(}\mathit{X}\mathbf{\right)}$	$\mathbf{\Delta }\overline{\mathit{X}}$	$\mathbf{\Delta }\mathit{var}\mathbf{\left(}\mathit{X}\mathbf{\right)}$	$\mathbf{\Delta }\overline{\mathit{X}}$	$\mathbf{\Delta }\mathit{var}\mathbf{\left(}\mathit{X}\mathbf{\right)}$
	Sarh
MaxTemp	−0.14 (0.55)	−0.15 (0.91)	−0.19 (0.70)	−0.47 (0.81)	−0.33 (0.47)	−0.62 (0.76)
MinTemp	−0.67 (0.35)	−13.76 (0.22)	−1.66 (0.29)	−4.97 (0.78)	−2.33(0.08)	−18.72 (0.33)
Precipitation	2.73 (0.72)	62.30 (0.53)	−0.12 (0.99)	−99.79 (0.88)	2.62 (0.86)	566.50 (0.86)
	Bongor
MaxTemp	−0.34 (0.73)	−3.90 (0.99)	−2.04 (0.31)	−5.68 (0.71)	−2.38 (0.20)	−9.58 (0.70)
MinTemp	−0.33 (0.72)	−11.75 (0.56)	−2 (0.31)	−1.26 (0.92)	−2.33 (0.19)	−13.01 (0.68)
Precipitation	−3.55 (0.66)	−1864 (0.63)	−2.62 (0.68)	850.40 (0.80)	−6.18 (0.88)	−1013.94 (0.57)
	Doba
MaxTemp	−0.14 (0.55)	−0.15 (0.92)	−0.19 (0.70)	−0.47 (0.81)	−0.33 (0.47)	−0.62 (0.76)
MinTemp	−6.67 (0.35)	−13.76 (0.22)	−1.66 (0.29)	−4.97 (0.78)	−2.33 (0.08)	−18.72 (0.33)
Precipitation	2.22 (0.81)	871.38 (0.83)	0.08 (0.99)	91.55 (0.96)	2.30 (0.89)	962.93 (0.95)
	Lai
MaxTemp	−1.90 (0.19)	−33.51 (0.52)	−3.78 (0.23)	−35.44 (0.47)	−5.68 (0.04)	−68.95 (0.24)
MinTemp	−1.92 (0.16)	−56.58 (0.15)	−3.65 (0.23)	−18.63 (0.74)	−5.57 (0.03)	−75.22 (0.25)
Precipitation	0.29 (0.96)	−14.31 (0.90)	−0.29 (0.98)	−243.09 (0.88)	−0.005 (0.99)	−257.39 (0.82)
	Ndjamena
MaxTemp	−0.36 (0.70)	15.2 (0.24)	−1.36 (0.45)	−1.64 (0.91)	−1.73 (0.35)	13.56 (0.61)
MinTemp	−0.39 (0.67)	8.92 (0.32)	−1.31 (0.45)	1.08 (0.99)	−1.70 (0.32)	10.01 (0.59)
Precipitation	−3.26 (0.62)	−843 (0.60)	−3.05 (0.82)	−45.27 (0.78)	−6.31 (0.60)	−888.28 (0.55)

). This significant difference in variance may be explained by the fact that the validation set coincides with a set of extreme values, caused by the drought that happened between 2006 and 2007, and which has as consequence to increase the bias of the calibrated model toward unseen data. This is a common issue encountered in the process of implementing a regression based model like the SDSM. Nevertheless, the values of NSE show that the models may be useful. In addition, the performances obtained are comparable with results generally observed in the literature [10,11,13,31,47]. For example, the average RMSE of the recorded literature survey is $72.62$ millimeters, which is greater than the average RMSE of $53.32$ millimeters, observed in each of the five gauging station considered in this case study.

The prediction performances of SDSM for minimum and maximum temperatures at the five gauging stations are given in Figure 6 and Figure 7, respectively. The performances obtained suggest an average RMSE of $8.68$ degree celsius, a MAPE of $17.40$ percent, a NSE of 0.64 as well as a coefficient of determination of $68.43$ percent on the validation set, for minimum temperatures forecast. For maximum temperatures, the results suggest an average RMSE of $9.93$ degree celsius, a mean MAPE of $14.55$ percent, a NSE of 0.47 and a coefficient of determination of $81.10$ percent on the validation set. These results are indicators of the robustness of the calibrated model developed for minimum and maximum temperatures forecast and fall within the threshold of what is generally encountered in the literature [13,20,48,49].

Nevertheless, it is worthwhile to note the descrepency of the model performances, between the calibration and the validation sets. This may be due to both the difference in the variance between the calibration and the validation sets (see Table 5), as well as the structural difference of the data, in the two subsets, as shown by the boxplot, in Figure 8. Additionally, this may suggest that SDSM is sensitive to temporal heterogeneity in the dataset.

4.2. LSTM Results

The LSTM performances for the five gauging stations, as well as the average prediction performance in the LCB are presented in Figure 9, for precipitation. The performances obtained suggest an average RMSE of $40.69$, an average MAPE of $28.55$ percent and an average NSE of $0.82$ on the test set, and which shows reliable forecasting. This observation is consistent for each of the five (05) gauge station considered in the study. The minimum NSE is observed in Lai (NSE = 0.71), and its maximum value is recorded in Sarh (NSE = 0.98). The loss function plot shown in Figure 4, indicate that there is no overfitting in the LSTM models trained for precipitation forecasting, and the above performances show that reanalysis data are suitable predictors for precipitations forecasting.

The prediction performances for each station as well as the average performances in the Lake Chad Basin, for minimum and maximum temperatures, are presented in Figure 10 and Figure 11, exhibit a mean RMSE of $0.66$ degree celsius for minimum temperatures and $1.22$ degree celsius for maximum temperatures. The average MAPE informs that the prediction errors are $2.93$ percent for minimum temperatures and $2.74$ percent for maximum temperatures. In addition, the average NSE (0.95 and 0.97 for minimum and maximum temperature respectively), indicate that the LSTM models trained for minimum and maximum temperature forecasting are very suitable for practical use. Finally, the loss function plots shown in Figure 4, indicate that there is no overfitting in the LSTM models, trained for temperature forecasting. These results show that reanalysis data are suitable predictors for temperatures forecasting.

4.3. Forecasting Comparison

A sample of the actual and predicted precipitation values on the test set for Ndjamena are given in Figure 12. A sample of the actual and observed temperatures for Ndjamena are given in Figure 13 and Figure 14. The monthly prediction biases (errors), as well as the correlations between predicted and observed values are provided in

Table 6. Monthly predictions biases and monthly correlations between observed and predicted values.

	Precipitation
	Error		Correlation
	sdsm	lstm	sdsm	lstm
January	11.47	1.34	0.33	0.92
February	11.46	1.32	0.33	0.75
March	−3.22	−4.26	0.55	0.96
April	−8.11	−13.17	0.74	0.85
May	19.52	−1.48	0.25	0.77
June	34.57	13.93	0.65	0.83
July	72.84	37.81	0.01	0.84
August	28.89	25.63	0.40	0.83
September	3.23	−5.65	0.64	0.94
October	7.67	1.91	0.55	0.88
November	−1.28	−3.69	0.57	0.91
December	−1.95	−3.61	0.88	0.91
	Minimum Temperature
	Error		Correlation
	sdsm	lstm	sdsm	lstm
January	−1.30	0.12	0.36	0.82
February	−1.29	0.09	0.52	0.72
March	−0.90	0.45	0.63	0.74
April	−0.49	−0.39	0.08	0.89
May	−2.64	−0.29	0.21	0.90
June	−2.29	−0.07	0.77	0.76
July	−0.75	−0.22	0.04	0.74
August	−1.35	0.44	0.53	0.80
September	−0.16	0.60	0.86	0.87
October	0.53	0.74	0.25	0.95
November	−0.51	0.36	0.67	0.85
December	−1.73	0.68	0.96	0.92
	Maximum Temperature
	Error		Correlation
	sdsm	lstm	sdsm	lstm
January	2.38	0.77	0.01	0.89
February	2.30	1.18	0.25	0.93
March	2.53	1.87	0.01	0.90
April	6.83	3.01	0.05	0.74
May	6.69	2.25	0.24	0.75
June	1.51	0.01	0.56	0.77
July	2.20	0.03	0.59	0.75
August	1.09	−0.01	0.35	0.87
September	0.92	0.05	0.35	0.91
October	3.05	0.20	0.01	0.85
November	3.17	−0.20	0.06	0.75
December	1.51	1.69	0.91	0.97

. The monthly prediction errors, indicate a tendency of underestimation in the LSTM predictions in March, April, May, September, November and December, and a tendency of overestimation, for the months of January, February, June, July, August, and October for precipitations. With regards, to the minimum temperature, LSTM predictions are underestimated in April, May, June and July, and overestimated in the other months. Maximum temperatures are underestimated in August, and overestimated elsewhere. In addition, the correlation between the observed and the predicted values of temperatures and precipitation show that both are highly related, especially for the prediction obtained with LSTM.

5. Discussion

5.1. Performance of LTSM vs. SDSM

The multivariate LSTM for precipitations forecasting presents both the smallest RMSE, MAPE, and the highest NSE (see Figure 15). Therefore, LSTM may be a better prediction tool for long-term precipitations in the Lake Chad Basin.

A comparison of SDSM and LSTM for minimum and maximum temperatures forecasting per gauging station (see Figure 16 and Figure 17) indicate that LSTM produces better performances in all five gauging stations, with the smaller values of RMSE and MAPE, and the higher values of NSE, for all the five gauging stations (see Figure 16 and Figure 17).

In terms of maximum temperature, LSTM outperforms SDSM in the five gauging stations, showing smaller values of RMSE, MAPE as well as the higher values of NSE.

The RMSE results shows the better abilities of LSTM to handle outliers as compared to SDSM, and NSE results indicates better reliability of LSTM forecasting outputs, for practical use, related to water ressources management and other hydrological purposes.

The better performances of neural networks in comparison to SDSM for precipitations, and temperatures forecasting observed in this study, do not align well with the results reported by Harphan and Wilby [20]. They compared the performances of RNN and SDSM in some gauging stations in England. Results of previous research still establish SDSM as the benchmark approach. This paper presents an exception. The tendency of LSTM to outperform SDSM in the Lake Chad Basin case may suggests LSTM as a good alternative to SDSM. The broader literature also suggest that LSTM architectures generally present an additional advantage in the forecasting of time series [50,51]. Moreover, LSTM performances can be improved through an automated hyperparameters search, with an advanced algorithms such as grid search or Bayesian optimisation, which are not investigated in this paper.

5.2. The Spatial and Temporal Heterogeneity of Microclimates in Forecasting Precipitation and Temperature

This study has shown the performance of SDSM and LSTM to be dependent on the local microclimates within the Lake Chad Basin. This application showed that SDSM had a higher NSE for stations with a higher overall MAP (Ndjamena 500–750 mm/yr) compared to stations with a lower MAP (Sarh, Doba, Bongor, Lai: 1000–1500 mm/yr).

For LSTM the largest NSE was less dependent on stations’ MAP. In the minimum and maximum monthly temperature forecasts, SDSM tended to perform worse for Doba in terms of NSE, where this station had a lower average monthly temperature (25–27.5 degree celsius) compared to stations in Sarh, Bongor, Lai and Ndjamena (27.5–30.3 degree celsius). In contrast, LSTM performance was less biased by the overall average temperature and was less dependent on local microclimate in this study. From the above observations, LSTM may present better abilities to handle the spatial heterogeneity of microclimates in forecasting precipitation and temperature, in the Lake Chad Basin. To fully evaluate the advantage of LSTM over SDSM for conditions with higher local microclimate variability, catchments with more in-situ data need to be selected. The application of LSTM has potential relevance for forecasting temperature and precipitation in regions with known high microclimatic variability, such as the Berg River in the Western Cape of South Africa [52], but also a region experiencing localised droughts [53]. Forecasting and the bias correction of GCM products using LSTM has the potential to be useful where recent data has shown spatial precipitation shifts and where other approaches tend to be more bias for local microclimates.

6. Conclusions

The aim of this paper was to compare the performances of SDSM and LSTM for monthly time–scales precipitations and temperatures forecasting in the Lake Chad basin, using grid resolution GCM output at a 5 degrees latitude × 5 degrees longitude global grid. This was motivated by the absence of such comparison in the literature. MLP and classical RNN were previously investigated in the literature, and results still confirmed SDSM as benchmark method for monthly precipitations and temperatures forecasting. From the results obtained in this study, the following conclusions can be made:

• LSTM provides more accurate monthly precipitations and temperatures forecasting outputs than SDSM, both in terms of sensitivity to outliers, with smaller RMSE, or in terms of forecasting reliability, with higher NSE coefficients;
• SDSM monthly precipitations and temperatures forecasting outputs are subject to the spatial heterogeneity of microclimates. This leads to less accurate forecasting outputs, in stations with higher MAP. Yet LSTM is capable to consider those variability, in the forecasting process;
• The comparison of LSTM with SDSM show that LSTM has high potential for monthly forecasting and the bias correction of GCM products, where recent data has shown spatial precipitation shifts and where other approaches tend to be more biased for local microclimates.
• The performances of LSTM could be enhanced by adding hyperparameter optimisation methods, such as grid search, and Bayesian optimisation. This may contribute further to establish LSTM as a benchmark for monthly climate forecasting.