# Influence of Random Forest Hyperparameterization on Short-Term Runoff Forecasting in an Andean Mountain Catchment

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

^{2}, %Bias, and RMSE metrics. We found that: (i) The most influencing hyperparameter is the number of trees in the forest, however the combination of the depth of the tree and the number of features hyperparameters produced the highest variability-instability on the models. (ii) Hyperparameter optimization significantly improved model performance for higher lead times (12- and 24-h). For instance, the performance of the 12-h forecasting model under default RF hyperparameters improved to R

^{2}= 0.41 after optimization (gain of 0.17). However, for short lead times (4-h) there was no significant model improvement (0.69 < R

^{2}< 0.70). (iii) There is a range of values for each hyperparameter in which the performance of the model is not significantly affected but remains close to the optimal. Thus, a compromise between hyperparameter interactions (i.e., their values) can produce similar high model performances. Model improvements after optimization can be explained from a hydrological point of view, the generalization ability for lead times larger than the concentration time of the catchment tend to rely more on hyperparameterization than in what they can learn from the input data. This insight can help in the development of operational early warning systems.

## 1. Introduction

## 2. Materials and Methods

#### 2.1. Study Site

^{2}catchment with elevations ranging from 2600 to 4200 m above sea level (m a.s.l.) (Figure 1). The mean annual rainfall is 850 and 1100 mm in the lower and upper parts of the catchment, respectively. Two rainy seasons have been identified, a first one during March-April-May, and a second shorter one in October. The importance of the Tomebamba catchment is related to its water-supplier role for Cuenca. However, the catchment is also responsible for periodically flooding parts of the city.

#### 2.2. Instruments and Data

#### 2.3. Random Forest and Hyperparameters

#### 2.4. Models Configuration

#### 2.5. Sensitivity Analysis of Hyperparameters

^{2}metric. Thus, we selected the hyperparameters that highly varied from their default value when reaching the highest model performances. This allowed us to identify the most relevant hyperparameters that needed a more detailed inspection.

^{2}metric calculated between observed and simulated data [39]. As this is an exhaustive search method, it was necessary to ensure that the optimal values of the hyperparameters were within the defined vectors [39]. An approach of 3-fold cross-validation was used in this study, which is a standard for evaluating the error in RF models.

#### 2.6. Performance Evaluation Criteria

^{2}), the root mean square error (RMSE), and percentage bias (%Bias) according to equations 1, 2, and 3, respectively.

## 3. Results

#### 3.1. Models Configuration: Inputs Setup

#### 3.2. Sensitivity Analysis of Hyperparameters

^{2}correspond to the maximum number of features (max_features: inputs to the model), the number of estimators (n_estimators: number of trees in the forest) and the maximum depth of the tree (max_depth: consecutive splits of a tree). Therefore, the subsequent analyzes were carried out only with these hyperparameters.

^{2}for each of the possible combinations. Figure 5 illustrates the evolution of the selected hyperparameters through the search space. Figure 5a,b depicts similar behaviors of the max_depth and n_estimators, where model efficiencies increase until reaching their maximum values. Thereafter, model performances remain almost constant (maximum variation of 0.03 for R

^{2}).

^{2}) when compared to the remaining hyperparameters However, after a threshold value is achieved (100 trees for the 12-h model), their efficiencies stabilized with a maximum variability of 0.01 for the training-validation subset.

^{2}was less than 0.03 for the models with max_depth values along the range of 10–70. The hyperparameter max_features presented the best performance (R

^{2}= 0.75) for a relatively low number of features (6). The model performance decreased until R

^{2}= 0.73 for 24 number of features, then increased again slightly for the value of the max_features = 30 and continued to increase slightly for higher values (less than 0.01). Thus the performance was relatively stable for the entire range of values analyzed.

#### 3.3. Optimization of Hyperparameters at Different Lead Times

^{2}) were 0.02, 0.16, and 0.14 for lead times of 4, 12, and 24 h, respectively. For instance, Figure 7 illustrates the scatter plot between the 12-h observed and predicted runoff using (a) default hyperparameters and (b) optimized hyperparameters. Results revealed the ability of the model to forecast runoffs up to 30 m

^{3}/s (20-year exceedance probability of less than 5%). Conversely, it seems clear the difficulty of the model to forecast extreme peak runoffs since a better representation of soil moisture may be required for modelling.

## 4. Discussion

## 5. Conclusions

- (i)
- Runoff forecasting model performance at 4 h denoted no significant difference in the R
^{2}metric by using both default and optimized hyperparameters. This is due to the relatively short lead-time forecasting that makes the input features become more relevant for the model. Therefore, even without performing a hyperparameter optimization process, a relatively high performance is obtained. On the other hand, for higher lead times (12- and 24-h) model performance is drastically improved when using optimized hyperparameters since they contribute more to the generalization of the model. - (ii)
- The importance of RF hyperparameterization was demonstrated in this study (high variability of solution surfaces). Thus, we suggest performing sensitivity analyses on input composition as well as on the most relevant RF hyperparameters for achieving optimal runoff forecasting efficiencies. This is especially true for lead times exceeding the concentration time of the associated catchment.
- (iii)
- The hyperparameter that causes the greater improvement in model performance when applying the optimization is the number of trees. Default value of n_estimators = 10 produced poor results. However, when its value increased to 100 and forward, model performance increase dramatically, especially in high lead times (12, 24). Thus, for a straightforward improvement in the performance of runoff RF models we recommended setting the n_estimators hyperparameter higher to 100.
- (iv)
- The hyperparameter max_depth produced the highest variability-instability on the models. Although its impact also depends on the combination with the max_features hyperparameter, it seems that the depth of the tree plays a key role on the generalization capability of the models. This is obvious when considering that it allows the model to generate specific solutions. However, finding that several combinations of max_features only produce slight variations in the model performance allows the modeler to focus on finding an optimal solution for max_depth only. This will reduce the computing times in the training-validation phase and would allow for a deeper exploration regarding the input features.

## Author Contributions

## Funding

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## References

- Breiman, L. Random Forrests. Mach. Learn.
**2001**, 45, 5–32. [Google Scholar] [CrossRef][Green Version] - Goel, E.; Abhilasha, E. Random Forest: A Review. Int. J. Adv. Res. Comput. Sci. Softw. Eng.
**2017**, 7, 251–257. [Google Scholar] [CrossRef] - Tyralis, H.; Papacharalampous, G.; Langousis, A. A brief review of random forests for water scientists and practitioners and their recent history in water resources. Water
**2019**, 11, 910. [Google Scholar] [CrossRef][Green Version] - Booker, D.J.; Snelder, T.H. Comparing methods for estimating flow duration curves at ungauged sites. J. Hydrol.
**2012**, 434, 78–94. [Google Scholar] [CrossRef] - Gislason, P.O.; Benediktsson, J.A.; Sveinsson, J.R. Random forests for land cover classification. Pattern Recognit. Lett.
**2006**, 27, 294–300. [Google Scholar] [CrossRef] - Puissant, A.; Rougiera, S.; Stumpf, A. Object-oriented mapping of urban trees using random forestclassifiers. Int. J. Appl. Earth Obs. Geoinf.
**2014**, 26, 235–245. [Google Scholar] [CrossRef] - Wang, L.; Zhou, X.; Zhu, X.; Dong, Z.; Guo, W. Estimation of biomass in wheat using random forest regression algorithm and remote sensing data. Crop J.
**2016**, 4, 212–219. [Google Scholar] [CrossRef][Green Version] - Naghibi, S.A.; Pourghasemi, H.R.; Dixon, B. GIS-based groundwater potential mapping using boosted regression tree, classification and regression tree, and random forest machine learning models in Iran. Environ. Monit. Assess.
**2016**, 188, 1–27. [Google Scholar] [CrossRef] - Liaw, A.; Wiener, M. Classification and Regression by randomForest. R News
**2002**, 2, 18–22. [Google Scholar] - Nitze, I.; Schulthess, U.; Asche, H. Comparison of machine learning algorithms random forest, artificial neural network and support vector machine to maximum likelihood for supervised crop type classification. In Proceedings of the 4th GEOBIA, Rio de Janeiro, Brazil, 7–9 May 2012; p. 35. [Google Scholar]
- Diez-Sierra, J.; del Jesus, M. Subdaily rainfall estimation through daily rainfall downscaling using random forests in Spain. Water
**2019**, 11, 125. [Google Scholar] [CrossRef][Green Version] - Seni, G.; Elder, J.F. Ensemble Methods in Data Mining: Improving Accuracy Through Combining Predictions. Synth. Lect. Data Min. Knowl. Discov.
**2010**, 2, 1–126. [Google Scholar] [CrossRef] - Louppe, G.; Wehenkel, L.; Sutera, A.; Geurts, P. Understanding variable importances in Forests of randomized trees. In Proceedings of the 26th International Conference on Neural Information Processing Systems; Curran Associates Inc.: Red Hook, NY, USA, 2013; Volume 1, pp. 431–439. [Google Scholar]
- Probst, P.; Wright, M.N.; Boulesteix, A.L. Hyperparameters and tuning strategies for random forest. Wiley Interdiscip. Rev. Data Min. Knowl. Discov.
**2019**, 9, e1301. [Google Scholar] [CrossRef][Green Version] - Van Rijn, J.N.; Hutter, F. An empirical study of hyperparameter importance across datasets. CEUR Workshop Proc.
**2017**, 1998, 91–98. [Google Scholar] - Probst, P.; Boulesteix, A.-L. To tune or not to tune the number of trees in random forest. J. Mach. Learn. Res.
**2017**, 18, 6673–6690. [Google Scholar] - Feng, Y.; Cui, N.; Gong, D.; Zhang, Q.; Zhao, L. Evaluation of random forests and generalized regression neural networks for daily reference evapotranspiration modelling. Agric. Water Manag.
**2017**, 193, 163–173. [Google Scholar] [CrossRef] - Fathian, F.; Mehdizadeh, S.; Kozekalani Sales, A.; Safari, M.J.S. Hybrid models to improve the monthly river flow prediction: Integrating artificial intelligence and non-linear time series models. J. Hydrol.
**2019**, 575, 1200–1213. [Google Scholar] [CrossRef] - Muñoz, P.; Orellana-Alvear, J.; Willems, P.; Célleri, R. Flash-flood forecasting in an andean mountain catchment-development of a step-wise methodology based on the random forest algorithm. Water
**2018**, 10, 1519. [Google Scholar] [CrossRef][Green Version] - Bond, N.R.; Kennard, M.J. Prediction of Hydrologic Characteristics for Ungauged Catchments to Support Hydroecological Modeling. Water Resour. Res.
**2017**, 53, 8781–8794. [Google Scholar] [CrossRef] - Erechtchoukova, M.G.; Khaiter, P.A.; Saffarpour, S. Short-Term Predictions of Hydrological Events on an Urbanized Watershed Using Supervised Classification. Water Resour. Manag.
**2016**, 30, 4329–4343. [Google Scholar] [CrossRef] - Orellana-Alvear, J.; Célleri, R.; Rollenbeck, R.; Muñoz, P.; Contreras, P.; Bendix, J. Assessment of native radar reflectivity and radar rainfall estimates for discharge forecasting in mountain catchments with a random forest model. Remote Sens.
**2020**, 12, 1986. [Google Scholar] [CrossRef] - Li, M.; Zhang, Y.; Wallace, J.; Campbell, E. Estimating annual runoff in response to forest change: A statistical method based on random forest. J. Hydrol.
**2020**, 589, 125168. [Google Scholar] [CrossRef] - Zhang, Y.; Chiew, F.H.S.; Li, M.; Post, D. Predicting runoff signatures using regression and hydrological modeling approaches. Water Resour. Res.
**2018**, 54, 7859–7878. [Google Scholar] [CrossRef] - Papacharalampous, G.A.; Tyralis, H. Evaluation of random forests and Prophet for daily streamflow forecasting. Adv. Geosci.
**2018**, 45, 201–208. [Google Scholar] [CrossRef][Green Version] - Wang, Z.; Lai, C.; Chen, X.; Yang, B.; Zhao, S.; Bai, X. Flood hazard risk assessment model based on random forest. J. Hydrol.
**2015**, 527, 1130–1141. [Google Scholar] [CrossRef] - Cai, J.; Luo, J.; Wang, S.; Yang, S. Feature selection in machine learning: A new perspective. Neurocomputing
**2018**, 300, 70–79. [Google Scholar] [CrossRef] - Jaiswal, J.K.; Samikannu, R. Application of Random Forest Algorithm on Feature Subset Selection and Classification and Regression. In Proceedings of the 2nd World Congress on Computing and Communication Technologies, WCCCT 2017, Tamil Nadu, India, 2–4 February 2017. [Google Scholar]
- Bernard, S.; Heutte, L.; Adam, S.; Bernard, S.; Heutte, L.; Adam, S. Influence of Hyperparameters on Random Forest Accuracy. In International Workshop on Multiple Classifier Systems; Springer: Berlin/Heidelberg, Germany, 2009; pp. 171–180. [Google Scholar]
- Fernández-Delgado, M.; Cernadas, E.; Barro, S.; Amorim, D. Do we need hundreds of classifiers to solve real world classification problems? J. Mach. Learn. Res.
**2014**, 15, 3133–3181. [Google Scholar] - Huang, B.F.F.; Boutros, P.C. The parameter sensitivity of random forests. BMC Bioinformatics
**2016**, 300, 70–79. [Google Scholar] [CrossRef][Green Version] - Liu, D.; Fan, Z.; Fu, Q.; Li, M.; Faiz, M.A.; Ali, S.; Li, T.; Zhang, L.; Khan, M.I. Random forest regression evaluation model of regional flood disaster resilience based on the whale optimization algorithm. J. Clean. Prod.
**2020**, 250, 119468. [Google Scholar] [CrossRef] - Bergstra, J.; Bengio, Y. Random search for hyper-parameter optimization. J. Mach. Learn. Res.
**2012**, 13, 281–305. [Google Scholar] - Berezowski, T.; Chybicki, A. High-resolution discharge forecasting for snowmelt and rainfall mixed events. Water
**2018**, 10, 56. [Google Scholar] [CrossRef][Green Version] - Gordon, A.D.; Breiman, L.; Friedman, J.H.; Olshen, R.A.; Stone, C.J. Classification and Regression Trees. Biometrics
**1984**, 1, 14–23. [Google Scholar] [CrossRef][Green Version] - Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; et al. Scikit-learn: Machine learning in Python. J. Mach. Learn. Res.
**2011**, 12, 2825–2830. [Google Scholar] - Sudheer, K.P.; Gosain, A.K.; Ramasastri, K.S. A data-driven algorithm for constructing artificial neural network rainfall-runoff models. Hydrol. Process.
**2002**, 16, 1325–1330. [Google Scholar] [CrossRef] - Brockwell, P.J.; Davis, R.A. Introduction to Time Series and Forecasting, 2nd ed.; Springer: Berlin/Heidelberg, Germany, 2002. [Google Scholar]
- Ataei, M.; Osanloo, M. Using a Combination of Genetic Algorithm and the Grid Search Method to Determine Optimum Cutoff Grades of Multiple Metal Deposits. Int. J. Surf. Mining, Reclam. Environ.
**2003**, 18, 60–78. [Google Scholar] [CrossRef]

**Figure 3.**(

**a**) Temporal autocorrelation and (

**b**) partial autocorrelation function of the runoff time series. The blue hatch indicates the 95% confidence band.

**Figure 4.**Pearson’s correlation between the five precipitation time series (one for each region) and the runoff time series.

**Figure 5.**Evolution of the 12-h forecast model performance (R

^{2}) along the different hyperparameter values in relation to (

**a**) the number of trees, (

**b**) maximum depth of the tree, and (

**c**) maximum number of features.

**Figure 6.**12-h forecast model performance (R

^{2}) for different combinations of max_depth and max_features for 3 cases of n_estimators (number of trees): (

**a**) 10, (

**b**) 50, and (

**c**) 200.

**Figure 7.**Observed vs. forecast runoff by using the test dataset (12 h lead-time). (

**a**) Default hyperparameters; (

**b**) optimized hyperparameters.

Hyperparameter | Description | Range | Default Value |
---|---|---|---|

criterion | The metric to measure the quality of a split | [mean absolute error, mean squared error] | mean squared error |

max_depth | The maximum depth that can reach a tree. | From 1 to number of training samples | ‘None’ (Until all leaves are pure) |

max_features | The maximum number of features that is allowed to try in individual tree. | From 1 to total number of features | Total number of features |

max_leaf_nodes | The maximum number of leaf nodes | From 1 to unlimited number of leaf nodes | Unlimited number of leaf nodes |

max_samples | The maximum number of samples to take to train each tree | From 1% to 100% | All samples |

min_samples_leaf | The minimum number of samples allowed to be a leaf node. | From 1 to total number of samples | 1 |

min_samples_split | The minimum number of samples allowed to split an internal node | From 2 to total number of samples | 2 |

n_estimators | The total number of trees in the forest. | From 1 to unlimited | 100 |

Hyperparameter | Value Vector; Increment |
---|---|

criterion | [mean absolute error, mean squared error] |

max_depth | [5–70;5] |

max_features | [6–48;6] |

max_leaf_nodes | [5–50;5] |

max_samples | [0.1–1;0.1] |

min_samples_leaf | [1–20;5] |

min_samples_split | [2, 5–40;5] |

n_estimators | [10–100;10, 100–1000;100] |

**Table 3.**Performance of the 4-, 12-, and 24-h runoff Random Forest (RF) models by using default hyperparameters and the test dataset.

Lead Time (Hours) | Training-Validation | Test | ||
---|---|---|---|---|

R^{2} | RMSE | %BIAS | R^{2} | |

4 | 0.75 | 6.14 | 9.88 | 0.70 |

12 | 0.68 | 9.36 | 31.22 | 0.24 |

24 | 0.48 | 9.85 | 28.15 | 0.16 |

**Table 4.**Optimal combination of RF hyperparameters for the 4-, 12- and 24-h runoff models and their corresponding performances.

Lead Time (Hours) | Optimal Hyperparameters | Training-Validation | Test | ||||
---|---|---|---|---|---|---|---|

n_estimators | max_features | max_depth | R^{2} | RMSE | %BIAS | R^{2} | |

4 | 500 | 42 | 30 | 0.85 | 5.96 | 10.12 | 0.69 |

12 | 500 | 6 | 30 | 0.75 | 8.26 | 24.33 | 0.41 |

24 | 500 | 6 | 35 | 0.54 | 9.01 | 29.02 | 0.29 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Contreras, P.; Orellana-Alvear, J.; Muñoz, P.; Bendix, J.; Célleri, R. Influence of Random Forest Hyperparameterization on Short-Term Runoff Forecasting in an Andean Mountain Catchment. *Atmosphere* **2021**, *12*, 238.
https://doi.org/10.3390/atmos12020238

**AMA Style**

Contreras P, Orellana-Alvear J, Muñoz P, Bendix J, Célleri R. Influence of Random Forest Hyperparameterization on Short-Term Runoff Forecasting in an Andean Mountain Catchment. *Atmosphere*. 2021; 12(2):238.
https://doi.org/10.3390/atmos12020238

**Chicago/Turabian Style**

Contreras, Pablo, Johanna Orellana-Alvear, Paul Muñoz, Jörg Bendix, and Rolando Célleri. 2021. "Influence of Random Forest Hyperparameterization on Short-Term Runoff Forecasting in an Andean Mountain Catchment" *Atmosphere* 12, no. 2: 238.
https://doi.org/10.3390/atmos12020238