# Evaluation of Four GLUE Likelihood Measures and Behavior of Large Parameter Samples in ISPSO-GLUE for TOPMODEL

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Materials and Methods

#### 2.1. Study Area and Data

^{2}. There are no major waterbodies and man-made dam structures within the watershed except a couple of small lakes on tributaries. There is the U.S. Geological Survey (USGS) streamflow gage 08041500 at the outlet, which has daily streamflow data from 1 January 2002 to 31 December 2013 [27]. Daily precipitation (PRCP) and pan evaporation (EVAP) data for the same period were available from the Global Historical Climatology Network-Daily (GHCND) dataset [28]. We used this pan evaporation data, which represents potential evapotranspiration [29], as an input to the model. Daily weather data from multiple weather stations (seven rainfall gages and two evapotranspiration gages) were combined by taking the area-weighted average based on the Voronoi diagram [30]. We split the entire data from 2002 to 2013 into five sets of calibration and validation periods to evaluate the impact of the simulation period on the performance of ISPSO-GLUE and GLUE. The calibration and validation periods are 2–6 years starting from 2002 and 2008, respectively, including the first year as a warm-up period (i.e., 2002 and 2008). Actual model evaluation was conducted starting from 2003 and 2009 without the warm-up period, hence 1–5 years of calibration and validation. The mean annual averages of precipitation and pan evaporation during the 12-year period are 997 mm and 1308 mm, respectively. The mean annual runoff ratio is 0.31. Based on the National Land Cover Database (NLCD) [31], the watershed consists of 0.3% open water, 6.3% developed land, 0.1% barren land, 45.0% forest, 32.4% shrub/grass/pasture, and 15.9% wetlands.

#### 2.2. Isolated-Speciation-Based Particle Swarm Optimization (ISPSO)

#### 2.3. ISPSO-Generalized Likelihood Uncertainty Estimation (GLUE)

#### 2.4. Topography Model (TOPMODEL)

#### 2.5. r.topmodel

#### 2.6. Four Likelihood Measures in GLUE

#### 2.6.1. Limits of Acceptability

#### 2.6.2. Absolute Mode Residual Methods

#### 2.6.3. Random Sampling and Simulation Periods

#### 2.7. Comparison of ISPSO-GLUE and GLUE Using the Nash–Sutcliffe (NS) Coefficient

## 3. Results and Discussion

#### 3.1. Limits of Acceptability and Absolute Model Residual Methods

#### 3.2. Wall-Clock Time of Simulations

#### 3.3. Model Performance

^{3}/d) is more than two times that for the 5-year validation period (1,124,422 m

^{3}/d). Also, the variability of the observed flow in the calibration period is much higher than in the validation period. These two simulation periods exhibit very different characteristics of the observed time series and those models that performed well in the calibration period have failed to show similar performance in the validation period for this reason. However, running all half a million random models—not just those behavioral models from the calibration period—in the validation period has produced the maximum NS coefficient of 0.45 for the GLUE method, which is smaller than the threshold NS value of 0.6. In other words, for the GLUE method, if we used the validation period (2009–2013) for calibration, we would have obtained no behavioral models at all from half a million random samples and there would be no behavioral models left to simulate the validation period (2003–2007). We performed a separate set of optimization runs using ISPSO-GLUE for the validation period and obtained the maximum NS coefficient of 0.45. This result confirms that the performance of TOPMODEL in the validation period may not exceed the threshold NS coefficient of 0.6.

#### 3.4. Behavior of Parameter Samples

#### 3.5. Uncertainty Bounds

#### 3.6. Suggestions

## 4. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Beven, K. A Manifesto for the Equifinality Thesis. J. Hydrol.
**2006**, 320, 18–36. [Google Scholar] [CrossRef] - ASME PTC Committee. Standard for Verification and Validation in Computational Fluid Dynamics and Heat Transfer; American Society of Mechanical Engineers: New York, NY, USA, 2009. [Google Scholar]
- Gorgoglione, A.; Bombardelli, F.A.; Pitton, B.J.; Oki, L.R.; Haver, D.L.; Young, T.M. Uncertainty in the Parameterization of Sediment Build-Up and Wash-Off Processes in the Simulation of Sediment Transport in Urban Areas. Environ. Model. Softw.
**2019**, 111, 170–181. [Google Scholar] [CrossRef] - Beven, K. On the Concept of Model Structural Error. Water Sci. Technol.
**2005**, 52, 167–175. [Google Scholar] [CrossRef] [PubMed] - Beven, K.; Binley, A. GLUE: 20 Years on. Hydrol. Process.
**2014**, 28, 5897–5918. [Google Scholar] [CrossRef] - Beven, K.; Binley, A. The Future of Distributed Models: Model Calibration and Uncertainty Prediction. Hydrol. Process.
**1992**, 6, 279–298. [Google Scholar] [CrossRef] - Freer, J.; Beven, K.; Ambroise, B. Bayesian Estimation of Uncertainty in Runoff Prediction and the Value of Data: An Application of the GLUE Approach. Water Resour. Res.
**1996**, 32, 2161–2173. [Google Scholar] [CrossRef] - Aronica, G.; Hankin, B.; Beven, K. Uncertainty and Equifinality in Calibrating Distributed Roughness Coefficients in a Flood Propagation Model with Limited Data. Adv. Water Resour.
**1998**, 22, 349–365. [Google Scholar] [CrossRef] - Beven, K.; Freer, J.; Hankin, B.; Schulz, K. The use of Generalised Likelihood Measures for Uncertainty Estimation in High Order Models of Environmental Systems. In Nonlinear and Nonstationary Signal Processing; Fitzgerald, W.J., Smith, R.L., Walden, A.T., Young, P.C., Eds.; Cambridge University Press: Cambridge, UK, 2000; pp. 115–151. ISBN 978-052-180-044-0. [Google Scholar]
- Beven, K.; Freer, J. Equifinality, Data Assimilation, and Uncertainty Estimation in Mechanistic Modelling of Complex Environmental Systems using the GLUE Methodology. J. Hydrol.
**2001**, 249, 11–29. [Google Scholar] [CrossRef] - Christiaens, K.; Feyen, J. Constraining Soil Hydraulic Parameter and Output Uncertainty of the Distributed Hydrological MIKE SHE Model using the GLUE Framework. Hydrol. Process.
**2002**, 16, 373–391. [Google Scholar] [CrossRef] - Makowski, D.; Wallach, D.; Tremblay, M. Using a Bayesian Approach to Parameter Estimation; Comparison of the GLUE and MCMC Methods. Agronomie
**2002**, 22, 191–203. [Google Scholar] [CrossRef] - Freer, J.E.; McMillan, H.; McDonnell, J.; Beven, K. Constraining Dynamic TOPMODEL Responses for Imprecise Water Table Information using Fuzzy Rule Based Performance Measures. J. Hydrol.
**2004**, 291, 254–277. [Google Scholar] [CrossRef] - Muleta, M.K.; Nicklow, J.W. Sensitivity and Uncertainty Analysis Coupled with Automatic Calibration for a Distributed Watershed Model. J. Hydrol.
**2005**, 306, 127–145. [Google Scholar] [CrossRef] - Zheng, Y.; Keller, A.A. Uncertainty Assessment in watershed-scale Water Quality Modeling and Management: 1. Framework and Application of Generalized Likelihood Uncertainty Estimation (GLUE) Approach. Water Resour. Res.
**2007**, 43, W08407. [Google Scholar] [CrossRef] - Smith, P.; Beven, K.J.; Tawn, J.A. Informal Likelihood Measures in Model Assessment: Theoretic Development and Investigation. Adv. Water Resour.
**2008**, 31, 1087–1100. [Google Scholar] [CrossRef] - Beven, K.; Smith, P. Concepts of Information Content and Likelihood in Parameter Calibration for Hydrological Simulation Models. J. Hydrol. Eng.
**2014**, 20, A4014010. [Google Scholar] [CrossRef] - Khu, S.; Werner, M.G. Reduction of Monte-Carlo Simulation Runs for Uncertainty Estimation in Hydrological Modelling. Hydrol. Earth Syst. Sci. Discuss.
**2003**, 7, 680–692. [Google Scholar] [CrossRef] - Blasone, R.; Vrugt, J.A.; Madsen, H.; Rosbjerg, D.; Robinson, B.A.; Zyvoloski, G.A. Generalized Likelihood Uncertainty Estimation (GLUE) using Adaptive Markov Chain Monte Carlo Sampling. Adv. Water Resour.
**2008**, 31, 630–648. [Google Scholar] [CrossRef] - Cho, H.; Olivera, F. Application of Multimodal Optimization for Uncertainty Estimation of Computationally Expensive Hydrologic Models. J. Water Resour. Plan. Man.
**2012**, 140, 313–321. [Google Scholar] [CrossRef] - Cho, H.; Kim, D.; Olivera, F.; Guikema, S.D. Enhanced Speciation in Particle Swarm Optimization for Multi-Modal Problems. Eur. J. Oper. Res.
**2011**, 213, 15–23. [Google Scholar] [CrossRef] - Arnold, J.G.; Srinivasan, R.; Muttiah, R.S.; Williams, J.R. Large Area Hydrologic Modeling and Assessment Part I: Model Development 1. J. Am. Water Resour. Assoc.
**1998**, 34, 73–89. [Google Scholar] [CrossRef] - Nash, J.E.; Sutcliffe, J.V. River Flow Forecasting through Conceptual Models Part I—A Discussion of Principles. J. Hydrol.
**1970**, 10, 282–290. [Google Scholar] [CrossRef] - Cho, H.; Kim, D.; Lee, K. Efficient Uncertainty Analysis of TOPMODEL using Particle Swarm Optimization. J. Korea Water Resour. Assoc.
**2014**, 47, 285–295. [Google Scholar] [CrossRef] - Beven, K.J.; Kirkby, M.J. A Physically Based, Variable Contributing Area Model of Basin hydrology/Un Modèle à Base Physique De Zone d’Appel Variable De l’Hydrologie Du Bassin Versant. Hydrol. Sci. J.
**1979**, 24, 43–69. [Google Scholar] [CrossRef] - Institute of Electrical and Electronics Engineers (IEEE); The Open Group. The Open Group Base Specifications Issue 7, 2018 Edition, IEEE Std 1003.1-2017 (Revision of IEEE Std 1003.1-2008). 2018. Available online: https://pubs.opengroup.org/onlinepubs/9699919799/ (accessed on 5 December 2018).
- U.S. Geological Survey (USGS). Surface-Water Daily Data for the Nation. 2016. Available online: http://waterdata.usgs.gov/nwis/sw (accessed on 10 March 2016).
- National Oceanic & Atmospheric Administration-National Climatic Data Center (NOAA-NCDC). Global Historical Climatology Network-Daily. 2016. Available online: http://www.ncdc.noaa.gov/cdo-web/search?datasetid=GHCND (accessed on 10 March 2016).
- Hobbins, M.T.; Ramírez, J.A. Trends in Pan Evaporation and Actual Evapotranspiration Across the Conterminous U.S.: Paradoxical or Complementary? Geophys. Res. Lett.
**2004**, 31, L13503. [Google Scholar] [CrossRef] - Voronoi, G. Nouvelles Applications Des Paramètres Continus à La Théorie Des Formes Quadratiques. Deuxième Mémoire. Recherches Sur Les Parallélloèdres Primitifs. J. Reine Angew. Math.
**1908**, 134, 198–287. [Google Scholar] [CrossRef] - U.S. Geological Survey (USGS). NLCD 2011 Land Cover (2011 Edition, amended 2014)—National Geospatial Data Asset (NGDA) Land Use Land Cover. 2014. Available online: https://www.mrlc.gov/data (accessed on 5 December 2018).
- U.S. Geological Survey (USGS). National Elevation Dataset (NED). 2016. Available online: http://nationalmap.gov/elevation.html (accessed on 10 March 2016).
- USDA-NRCS. Urban Hydrology for Small Watersheds, TR-55. 1986. Available online: https://www.nrcs.usda.gov/Internet/FSE_DOCUMENTS/stelprdb1044171.pdf (accessed on 16 March 2017).
- Cho, H.; Olivera, F. Effect of the Spatial Variability of Land use, Soil Type, and Precipitation on Streamflows in Small Watersheds 1. J. Am. Water Resour. Assoc.
**2009**, 45, 673–686. [Google Scholar] [CrossRef] - Eberhart, R.; Kennedy, J. A New Optimizer using Particle Swarm Theory. In Micro Machine and Human Science, MHS’95, Proceedings of the Sixth International Symposium, Nagoya, Japan, October 1995; IEEE: New York, NY, USA, 1995; pp. 39–43. [Google Scholar] [CrossRef]
- Kennedy, J.; Eberhart, R. Particle Swarm Optimization. In Proceedings of the Fourth IEEE International Conference on Neural Networks, Perth, WA, Australia, 27 November–1 December 1995; IEEE: New York, NY, USA, 1995; pp. 1942–1948. [Google Scholar]
- Li, X. Adaptively Choosing Neighbourhood Bests using Species in a Particle Swarm Optimizer for Multimodal Function Optimization. In Proceedings of the Genetic and Evolutionary Computation Conference, Seattle, WA, USA, 26–30 June 2004; pp. 105–116. [Google Scholar] [CrossRef]
- R Development Core Team. R: A Language and Environment for Statistical Computing; R Foundation for Statistical Computing: Vienna, Austria, 2006. Available online: http://www.r-project.org (accessed on 3 November 2015).
- Cho, H.; Yee, T.; Heo, J. Automated Floodway Determination using Particle Swarm Optimization. Water
**2018**, 10, 1420. [Google Scholar] [CrossRef] - Kim, D.; Olivera, F.; Cho, H. Effect of the Inter-Annual Variability of Rainfall Statistics on Stochastically Generated Rainfall Time Series: Part 1. Impact on Peak and Extreme Rainfall Values. Stoch. Environ. Res. Risk Assess.
**2013**, 27, 1601–1610. [Google Scholar] [CrossRef] - Kim, D.; Olivera, F.; Cho, H.; Lee, S.O. Effect of the Inter-Annual Variability of Rainfall Statistics on Stochastically Generated Rainfall Time Series: Part 2. Impact on Watershed Response Variables. Stoch. Environ. Res. Risk Assess.
**2013**, 27, 1611–1619. [Google Scholar] [CrossRef] - Kim, D.; Olivera, F.; Cho, H.; Socolofsky, S.A. Regionalization of the Modified Bartlett-Lewis Rectangular Pulse Stochastic Rainfall Model. Terr. Atmos. Ocean. Sci.
**2013**, 24. [Google Scholar] [CrossRef] - Kim, D.; Cho, H.; Onof, C.; Choi, M. Let-it-Rain: A Web Application for Stochastic Point Rainfall Generation at Ungaged Basins and its Applicability in Runoff and Flood Modeling. Stoch. Environ. Res. Risk Assess.
**2017**, 31, 1023–1043. [Google Scholar] [CrossRef] - Cho, H.; Kim, D.; Lee, K.; Lee, J.; Lee, D. Development and Application of a Storm Identification Algorithm that Conceptualizes Storms by Elliptical Shape. J. KOSHAM
**2013**, 13, 325–335. [Google Scholar] [CrossRef] - Heo, J.; Yu, J.; Giardino, J.R.; Cho, H. Impacts of Climate and land-cover Changes on Water Resources in a Humid Subtropical Watershed: A Case Study from East Texas, USA. Water Environ. J.
**2015**, 29, 51–60. [Google Scholar] [CrossRef] - Heo, J.; Yu, J.; Giardino, J.R.; Cho, H. Water Resources Response to Climate and Land-Cover Changes in a Semi-Arid Watershed, New Mexico, USA. Terr. Atmos. Ocean. Sci.
**2015**, 26. [Google Scholar] [CrossRef] - Van Griensven, A.; Meixner, T. A Global and Efficient Multi-Objective Auto-Calibration and Uncertainty Estimation Method for Water Quality Catchment Models. J. Hydroinform.
**2007**, 9, 277–291. [Google Scholar] [CrossRef] - Beven, K. Infiltration into a Class of Vertically Non-uniform Soils. Hydrol. Sci. J.
**1984**, 29, 425–434. [Google Scholar] [CrossRef] - Green, W.H.; Ampt, G.A. Studies on Soil Physics 1. The Flow of Air and Water Through Soils. J. Agric. Sci.
**1911**, 4, 11–24. [Google Scholar] [CrossRef] - Cho, H. A GIS Hydrological Modeling System by Using the Programming Interface of GRASS GIS. Master’s Thesis, Kyungpook National University, Daegu, Korea, 2000. [Google Scholar]
- Neteler, M.; Bowman, M.H.; Landa, M.; Metz, M. GRASS GIS: A Multi-Purpose Open Source GIS. Environ. Model. Softw.
**2012**, 31, 124–130. [Google Scholar] [CrossRef] - Buytaert, W. TOPMODEL R Package. 2009. Available online: https://source.ggy.bris.ac.uk/wiki/Topmodel (accessed on 3 November 2015).
- Conrad, O. SAGA-GIS Module Library Documentation (v2.1.3): Module TOPMODEL. 2003. Available online: http://www.saga-gis.org/saga_module_doc/2.1.3/sim_hydrology_2.html (accessed on 3 November 2015).
- Olaya, V. A Gentle Introduction to SAGA GIS; The SAGA User Group eV: Gottingen, Germany, 2004. [Google Scholar]
- Vrugt, J.A.; Diks, C.G.; Gupta, H.V.; Bouten, W.; Verstraten, J.M. Improved Treatment of Uncertainty in Hydrologic Modeling: Combining the Strengths of Global Optimization and Data Assimilation. Water Resour. Res.
**2005**, 41, W01017. [Google Scholar] [CrossRef] - Carr, D.B.; Littlefield, R.J.; Nicholson, W.; Littlefield, J. Scatterplot Matrix Techniques for Large N. J. Am. Stat. Assoc.
**1987**, 82, 424–436. [Google Scholar] [CrossRef]

**Figure 2.**Constant error limits of acceptability, simulated flows with the best Nash–Sutcliffe (NS), and 100 other rejected models. Rejected models are plotted first to clearly show the limits of acceptability. The limits of acceptability using the non-parametric error deviation are not shown because the width of the limits is very narrow.

**Figure 3.**The 95% uncertainty bounds for the 2003–2006 calibration period from May 2005 to November 2006 (wettest period with the highest peak flow). The shaded area and dotted line represent the uncertainty bounds and observed flows, respectively.

**Figure 4.**The 95% uncertainty bounds for the 2003–2007 calibration period from January 2006 to July 2007 (wettest period with the highest peak flow). The shaded area and dotted line represent the uncertainty bounds and observed flows, respectively.

**Figure 5.**Hexagonal bin plots of NS vs. model parameters for ISPSO-GLUE and GLUE for the 5-year calibration period. Darker hexagons represent a higher density of samples.

**Figure 6.**Convergence plot for the NS coefficient. The solid and dashed lines represent ISPSO-GLUE and GLUE, respectively. (

**a**) 2003 calibration, (

**b**) 2003 validation with all models, (

**c**) 2003–2004 calibration, (

**d**) 2003–2004 validation with all models, (

**e**) 2003–2005 calibration, (

**f**) 2003–2005 validation with all models, (

**g**) 2003–2006 calibration, (

**h**) 2003–2006 validation with all models, (

**i**) 2003–2007 calibration, (

**j**) 2003–2007 validation with all models.

Name | Description | Min | Max |
---|---|---|---|

qs0 | Initial subsurface flow per unit area in m/h | 0 | 0.0001 |

lnTe | Areal average of the soil surface transmissivity in ln(m^{2}/h) | −7 | 10 |

m | Scaling parameter describing the soil transmissivity in m | 0.001 | 0.25 |

Sr0 | Initial root zone storage deficit in m | 0 | 0.01 |

Srmax | Maximum root zone storage deficit in m | 0.005 | 0.08 |

Td | Unsaturated zone time delay per unit storage deficit in h | 0.001 | 40 |

vch | Main channel routing velocity in m/h | 50 | 2000 |

vr | Internal subcatchment routing velocity in m/h | 50 | 2000 |

K0 | Surface hydraulic conductivity in m/h | 0.0001 | 0.2 |

psi | Wetting front suction in m | 0.01 | 0.5 |

dtheta | Water content change across the wetting front | 0.01 | 0.6 |

**Table 2.**Number of behavioral models, percentage of enclosed observed flows, maximum NS, and number of model runs for the maximum NS. The threshold value for behavioral models is a NS coefficient of 0.6. Dashes indicate that there were no behavioral models for the calibration period and no simulations were performed for the validation period. For comparison purposes, all models including behavioral and non-behavioral models were run to calculate the maximum NS coefficient in the validation periods, which are displayed within parentheses.

Method | Simulation Period | Number of Behavioral Models | Percentage of Enclosed Observed Flows (%) | Maximum NS | Number of Model Runs for Maximum NS | |
---|---|---|---|---|---|---|

ISPSO-GLUE | Calibration | 2003 | 497,474 | 6.6 | 0.84 | 478 |

2003–2004 | - | - | 0.51 | 860 | ||

2003–2005 | - | - | 0.50 | 257,475 | ||

2003–2006 | 492,241 | 25.7 | 0.81 | 574 | ||

2003–2007 | 491,483 | 17.4 | 0.80 | 871 | ||

Validation | 2009 | 0 | 0 | 0.18 (0.28) | 290 (3676) | |

2009–2010 | - | - | - (0.37) | - (19,693) | ||

2009–2011 | - | - | - (0.39) | - (121) | ||

2009–2012 | 0 | 0 | 0.40 (0.42) | 183,567 (183,667) | ||

2009–2013 | 0 | 0 | 0.42 (0.42) | 71,723 (71,723) | ||

GLUE with random sampling | Calibration | 2003 | 6708 | 22.2 | 0.84 | 198,826 |

2003–2004 | - | - | 0.50 | 1550 | ||

2003–2005 | - | - | 0.49 | 17,579 | ||

2003–2006 | 28,716 | 30.9 | 0.81 | 1402 | ||

2003–2007 | 28,289 | 37.6 | 0.80 | 19,239 | ||

Validation | 2009 | 0 | 0 | 0.22 (0.31) | 320,279 (70,871) | |

2009–2010 | - | - | - (0.40) | - (346,578) | ||

2009–2011 | - | - | - (0.40) | - (1248) | ||

2009–2012 | 0 | 0 | 0.42 (0.44) | 153,849 (185,951) | ||

2009–2013 | 0 | 0 | 0.44 (0.45) | 79,442 (48,571) |

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Cho, H.; Park, J.; Kim, D. Evaluation of Four GLUE Likelihood Measures and Behavior of Large Parameter Samples in ISPSO-GLUE for TOPMODEL. *Water* **2019**, *11*, 447.
https://doi.org/10.3390/w11030447

**AMA Style**

Cho H, Park J, Kim D. Evaluation of Four GLUE Likelihood Measures and Behavior of Large Parameter Samples in ISPSO-GLUE for TOPMODEL. *Water*. 2019; 11(3):447.
https://doi.org/10.3390/w11030447

**Chicago/Turabian Style**

Cho, Huidae, Jeongha Park, and Dongkyun Kim. 2019. "Evaluation of Four GLUE Likelihood Measures and Behavior of Large Parameter Samples in ISPSO-GLUE for TOPMODEL" *Water* 11, no. 3: 447.
https://doi.org/10.3390/w11030447