Evaluating Hydrologic Model Performance for Characterizing Streamflow Drought in the Conterminous United States

Abstract

Hydrologic models are the primary tools that are used to simulate streamflow drought and assess impacts. However, there is little consensus about how to evaluate the performance of these models, especially as hydrologic modeling moves toward larger spatial domains. This paper presents a comprehensive multi-objective approach to systematically evaluating the critical features in streamflow drought simulations performed by two widely used hydrological models. The evaluation approach captures how well a model classifies observed periods of drought and non-drought, quantifies error components during periods of drought, and assesses the models’ simulations of drought severity, duration, and intensity. We apply this approach at 4662 U.S. Geological Survey streamflow gages covering a wide range of hydrologic conditions across the conterminous U.S. from 1985 to 2016 to evaluate streamflow drought using two national-scale hydrologic models: the National Water Model (NWM) and the National Hydrologic Model (NHM); therefore, a benchmark against which to evaluate additional models is provided. Using this approach, we find that generally the NWM better simulates the timing of flows during drought, while the NHM better simulates the magnitude of flows during drought. Both models performed better in wetter eastern regions than in drier western regions. Finally, each model showed increased error when simulating the most severe drought events.

1. Introduction

Drought is a costly natural disaster, sometimes causing billions of dollars’ worth of damage, and has wide-ranging impacts, from agriculture to public health [1,2,3]. While drought has received much attention in the hydrologic literature [4], drought is a complex phenomenon that is difficult to simulate [5,6,7,8,9]. Droughts are commonly separated into four categories: meteorological, agricultural (or soil moisture), hydrological, and socioeconomic [10,11,12]. Meteorological, agricultural, and hydrological droughts occur in physical systems, whereas socioeconomic droughts are the social and economic impacts of drought [12,13]. We focus on streamflow drought, which is a subset of hydrological drought, that Van Loon [11] defines as follows: “a lack of water in the hydrologic system manifesting itself in abnormally low streamflow in rivers.” Streamflow drought can be costly and negatively impact many sectors that rely on streamflow quantity, quality, and timing, such as ecosystem, agricultural, navigation, and municipal services [1].

Due to the importance of streamflow drought, there have been ongoing efforts to simulate and predict drought occurrence and severity [8,14,15,16,17,18,19]. Many studies and models have shown skill in simulating certain drought events; however, there are many different types of drought indicators [11,20,21], and the methodologies used to evaluate models are inconsistent [22]. Differences in the methodologies used to evaluate drought simulations can make model intercomparison difficult, as is the case more broadly in hydrology [23,24].

Streamflow drought is different to normal low flows. Although droughts may include periods of low streamflows, a recurring seasonal low-flow event is not necessarily a drought [25]. Numerous studies have aimed to quantify low flows (e.g., 7-day mean low-flow (Q7) [25]) and evaluate how well models simulate low-flow metrics [26,27,28,29,30], but these metrics alone are inadequate for evaluating model simulations of drought as they do not address the differences between low flows and drought, and often target a specific low-flow magnitude like Q7 instead of evaluating the full period of a drought.

Common model evaluation metrics, like the Nash–Sutcliffe efficiency (NSE; [31]), are often more sensitive to high flows than low flows, making them a poor indicator of predictive accuracy for drought [32]. To address this issue, drought studies sometimes use evaluation metrics on standardized drought indices (e.g., standardized streamflow index) or streamflow percentiles [11] instead of directly on the streamflow [6,33,34]. This offers some improvement, but often still focuses a large percentage of the evaluation on non-drought periods. Streamflow droughts are, by definition, abnormal events (often between the 2nd and 30th percentile [35,36]), meaning that, generally, 70–98% of streamflow data represent non-drought conditions. Applying metrics across non-drought periods may inflate or deflate model performance metrics, making them less indicative of a model’s performance when simulating drought [32].

Studies also use different methods of identifying drought events, making model intercomparison and benchmarking difficult. For example, drought can be identified with a fixed method (where a river is in a drought when it drops below a single fixed long-term level) or a variable method (where the drought threshold varies seasonally based on how much water is typically available in that season) [7,18,35,37].

Previous efforts to evaluate the simulation of streamflow drought vary substantially (e.g., using different temporal resolutions, periods or seasons of interest, methods for drought characterization, and different methods for evaluation), which makes model evaluation intercomparisons challenging. One reason for this is that the focus of traditional hydrologic modeling has been on local or regional catchment scales [23], where evaluations typically vary study by study, depending on the application or location. Similarly, studies sometimes focus on a particular drought event (e.g., [33,34,38]), which by design is not conducive to model evaluation intercomparisons. However, as hydrologic modeling moves to encompass national and larger spatial scales, and toward longer (multi-year) runs, there is a need to provide systematic, comprehensive, comparable metrics to evaluate the streamflow drought modeled across hydroclimatic regions. Previous model intercomparison studies have provided systematic evaluation approaches for Earth system models [39] and long-term streamflow performance [24], but Towler et al. [24] only looked at one low-flow metric.

Given the shift toward standardized approaches to the construction and evaluation of national-scale hydrologic models and the importance of streamflow drought, we propose a systematic and comprehensive approach to evaluating simulations of streamflow drought. We demonstrate the approach through an assessment of the streamflow drought simulation performance of two large-scale hydrological modeling applications in the conterminous United States (CONUS): the National Water Model version 2.1 application of WRF-Hydro (NWM; [40]; accessed through NOAA’s Office of Water Prediction) and the National Hydrologic Model application of the Precipitation-Runoff Modeling System version 1.0 three-step calibration with routing (NHM; [41,42,43]).

Specifically, we use three categories of metrics to evaluate model simulations of streamflow drought:

• Classification: How well the models simulate the occurrence of observed drought vs. non-drought periods [44] according to Cohen’s kappa [45]. Evaluating model simulation of drought occurrence is one of the simplest but most important measures of model performance in drought simulation.
• Error Components: How well the models simulate the timing, magnitude, and variability of streamflow during periods of drought according to Spearman’s r [46], percent bias, and the ratio of standard deviations, respectively. This approach examines how errors in streamflow are split across these three components [47].
• Drought Signatures: How well the models simulate drought duration, intensity, and severity according to the normalized mean absolute errors of annualized data. These three drought characteristics are widely used by the research community and play a large role in the impact of droughts.

This suite of metrics captures many facets of streamflow drought simulation and evaluates them across many hydrologic environments. This approach extends existing model evaluations of drought (e.g., [48]), with additional focus on multi-objective evaluation that emphasizes the critical features of streamflow drought which are relevant to understanding error. The evaluation of these models across a wider number and density of gages provides a more robust understanding of models [49] and importantly captures the heterogeneity in drought responses, which, especially in mountainous regions, can have major implications for regional responses and water security [50]. Applying this comprehensive evaluation in a benchmarking framework for the intercomparison of the NWM and NHM modeling application allows for an improved understanding of the differences and potential limitations and benefits of each model [24]. We apply advances from large sample intercomparison studies [24,49,51] to continue to build on studies examining streamflow drought simulations (e.g., [48]).

2. Materials and Methods

Prior to evaluating streamflow drought using the three categories of metrics, we acquired data at stream gages with sufficient historical records, identified stream gages in regions with similar hydroclimatic characteristics, and identified periods of streamflow drought that were suitable for the demonstration of the drought evaluation framework.

2.1. Modeling Applications and Observed Data

2.1.1. Modeling Applications

The NHM and NWM are similar in their temporal and spatial extent (CONUS-wide), in which the hydrologic state and flux variables are simulated, and in their use in hydrologic estimation and research. However, these modeling applications have some key differences, including the spatial and temporal resolution, parameter estimation techniques, parameter datasets, calibrations, and forcings. The NHM is forced with the 1 km Daymet version 3 product [52], whereas the NWM is forced with the NOAA-produced 1 km Analysis of Record for Calibration version 1.0 forcing dataset [53]. The NWM calculates states and fluxes on a 1 km grid, while the NHM uses hydrological response units and a stream network as the primary geospatial structure [24]. The NWM runs on an hourly timestep, while the NHM is daily.

While the model calibration details can be found in [24,43], here we highlight several key aspects of calibration for each model. The NWM calibrates its parameters for hourly streamflow at 1378 observation stations, mostly in natural (unimpaired) basins. The calibration uses a modified Nash–Sutcliffe efficiency (NSE; [31,54]), which includes the standard NSE, as well as a log-transformed NSE. Then, the NWM employs hydrologic similarity to regionalize the parameters for the remaining watersheds. On the other hand, the NHM considers multiple objectives in a stepwise manner in its calibration routine. First, it balances the water budgets in each hydrologic response unit (HRU), considers the streamflow timing based on a statistically generated dataset, and finally calibrates to the observed streamflow at 1417 gage locations.

Both the NWM and the NHM have been evaluated for their performance in simulating streamflow (e.g., [24,27,55,56,57,58]), but have not yet been evaluated against other models for simulating streamflow drought; they thus serve as good candidates for our methodology. We used daily mean streamflow simulations direct from the NHM and averaged the hourly streamflow simulations from the NWM into a daily mean streamflow for each gage of interest. A daily timestep was chosen for comparison to preserve as much of the event timing relationship as possible while still accommodating common modeling application timescales (many model applications simulate results at a daily timestep). Further descriptions of these modeling applications and calibration techniques are explained in Towler et al. [24] and Hay et al. [43].

2.1.2. Observed Data

The drought performance of both model applications was evaluated at 4662 U.S. Geological Survey (USGS) stream gages across CONUS. We subset the stream gage dataset for benchmarking the hydrologic modeling applications described in Foks et al. [59] to include gages with at least 16 years of daily observations of flow (a longer record was needed to obtain sufficient data during drought periods). The study period spans the climate years (CYs, April 1–March 31) 1985–2016, which is the overlap between the NWM and the NHM historical simulations. Climate years more consistently contain the entire annual low-flow period than calendar years (January through December) or water years (October through September; [60,61]) in CONUS.

2.1.3. Evaluating Regional Performance

We categorized the stream gages into 12 hydrologic regions (Figure 1), defined by their correlation in monthly flows among the minimally altered gages in the Hydro-Climatic Data Network (HCDN; [62]), following approaches by McCabe and Wolock [63]. We then evaluated the regional differences in model performance when simulating streamflow drought by examining the model performance across all stream gages in each region. These same regions were used in the regional drought analysis by Hammond et al. [35]. We attributed stream gages not within the HCDN to the region of the nearest HCDN gage.

2.2. Identification of Drought

We characterized drought using fixed and variable streamflow percentiles and thresholds at the 5th, 10th, 20th, and 30th percentiles, as implemented by Hammond et al. [35]. We converted the daily streamflow observations and model simulations to percentiles for all gages. Within observations or a single model, percentiles allow for the fair classification of drought across different hydroclimatic regions. Percentiles allow for more direct comparisons of low-flow anomalies (i.e., drought) between observations and models without the influence of model bias, which we evaluate elsewhere. The streamflow percentiles were computed with the Weibull plotting position (r/(n + 1), where r is rank and n is the number of data (e.g., [64])). Two types of percentile-based threshold approaches were used: (1) fixed—all modeled or observed flows in the period on record are used to calculate one fixed threshold, and (2) variable—unique thresholds are calculated for each day of the year using only the values for that day from all years on record (Figure S1a,b). We implemented a modified version [65] of the combined threshold level and continuous dry period methods developed by Van Huijgevoort et al. [66] to handle the zero-flow measurements (<0.00028 cubic meters per second; <0.01 cubic feet per second). This method breaks ties between zero-flow days for percentile rankings based on the number of preceding zero-flow days, with days with more preceding zero-flow days receiving lower percentile rankings. Droughts were classified according to whether the streamflow was below the 5th, 10th, 20th, or 30th percentiles (Figure S1c), which roughly correspond to the extreme, severe, moderate, or abnormal drought classifications used by the U.S. Drought Monitor (https://droughtmonitor.unl.edu/, accessed 1 November 2023). This created a time series of drought presence and absence for each stream gage, threshold, and simulated and observed streamflow dataset. For each climate year, we calculated the following:

• Drought duration: the total number of days below the threshold (days);
• Drought severity: the sum of flow departures or deficit below the threshold (cms-days);
• Drought intensity: the maximum drought intensity (minimum percentiles).

2.3. Evaluation for Drought Performance

We evaluate the performance of model simulations of drought by examining the goodness of the match [44] between modeled and observed events (i.e., event classification) and the quality of the simulation for matched events [44]. To evaluate the quality of the simulations during drought, we examine the error components during drought events (Spearman’s r, ratio of standard deviations, and percent bias) and the errors in the important characteristics of drought (drought signatures of duration, severity, and intensity) aggregated to the climate year level.

Table 1. Streamflow drought statistical metrics for daily streamflow drought evaluation for different metric categories. Additional calculation details can be found in Text S1.

Category	Statistic	Description	Range (Perfect)	Comments
Event Classification	Cohen’s kappa	Cohen’s kappa statistic for inter-rater reliability [45]	−1 to 1 (1)	A measure of agreement relative to the probability of achieving results by chance.
Error Components	Spearman’s r	Spearman’s rank correlation coefficient	−1 to 1 (1)	A nonparametric estimator of correlation for flow timing.
	Ratio of standard deviations	Ratio of simulated to observed standard deviations (for the scorecard this is presented as the absolute deviation from the target of 1)	0 to Inf (1)	Indicates if the flow variability is being over or underestimated.
	Percent bias	Percent bias (simulated minus observed) (for the scorecard this is presented as the absolute percent bias)	−100 to Inf (0)	Indicates if total streamflow volume is being over or underestimated.
Drought Signatures	Drought Duration	Normalized mean absolute error (NMAE) in the annual time series of drought duration, i.e., the sum of days of drought each year for a given threshold.	0 to Inf (0)	Indicates how well the model simulates annual drought durations.
	Drought Intensity	NMAE in the annual time series of the distance the minimum percentile is below the drought threshold, i.e., the overall maximum distance below the threshold for any drought during the year.	0 to Inf (0)	Indicates how well the model simulates annual minimum flow.
	Drought Severity (Flow Deficit Volume)	NMAE in the annual time series of drought deficit volume in cubic meters per second-days (cms-days), i.e., the sum of drought deficits for all droughts during the year.	0 to Inf (0)	Indicates how well the model simulates annual flow deficit. This is a measure of drought severity.

shows the streamflow drought statistical metrics included in our systematic evaluation. We evaluated performance in three categories— “Event Classification”, “Error Components”, and “Drought Signatures”—which are described in the subsections below.

2.3.1. Event Classification Evaluation

Drought and non-drought periods were first classified for modeled and observed time series data. Evaluating if the model can correctly classify drought is one of the simplest but most important measures of model performance in drought simulation. If a model cannot correctly simulate when a drought is occurring, the quality of the rest of its predictions in the context of drought may not be particularly useful. We used Cohen’s kappa to evaluate how well the simulated drought periods capture observed drought periods considering each day independently. Cohen’s kappa measures the accuracy of the model classification and accounts for class imbalance in cases where categorical results are not evenly balanced (i.e., for the 20th percentile drought threshold, 80% of days will be non-drought). Cohen’s kappa compares the relative observed agreement (true positives and true negatives from a contingency table or confusion matrix) to the expected agreement. Landis and Koch [67] provide guidelines for interpreting Cohen’s kappa. They describe values < 0 as indicating no agreement and 0–0.20 as slight, 0.21–0.40 as fair, 0.41–0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1 as almost perfect agreement [67]. Cohen’s kappa was calculated as follows:

$$k={\displaystyle \frac{p_o-p_e}{1-p_e}}$$

(1)

where $p_o$ is the relative observed agreement and $p_e$ is the expected agreement or probability of agreement.

2.3.2. Error Components Evaluation

Commonly used metrics of efficiency like NSE are aggregations of model error components, but more information and insights can often be obtained when these error components are decomposed [47,68,69]. Gupta et al. [47] decomposed NSE and showed that “NSE consists of three distinctive components representing the correlation, the bias, and a measure of relative variability in the simulated and observed value.” We evaluated these three individual components of model error rather than a single aggregated metric in order to provide insight into what might be driving model error during periods of drought. To evaluate the model performance during drought periods, we calculated the correlation (which shows errors from timing), bias (which shows errors in the magnitude of the streamflow distribution), and ratio of standard deviations (which evaluates errors in the variability of the streamflow distribution) of the simulated flows corresponding to the observed droughts. We followed a similar approach to that used by Pushpalatha et al. [32] (10th percentile of flow) and Pfannerstill et al. [30] (5th and 20th percentiles of flow) for evaluating the model performance for low flows and calculated our metrics only on values corresponding to drought events.

To evaluate the models’ ability to reproduce the sequence of the observed time series, in other words, the timing of streamflow, during drought periods, we calculated Spearman’s rank correlation coefficient [70,71]. Being based on the ranks of the flow magnitudes, Spearman’s r depends only on the monotonicity of the relation between the observed and simulated flows. As a result, Spearman’s r is a better estimator of the timing correlation than the more common Pearson estimator for streamflow data [72], since it is resistant to nonlinearity and skewness. Spearman’s r is often used to assess flow timing to determine how well a model reproduces the relative position in time of flow values [24,73]. Our calculations began in the same way as those for standard Spearman’s r, where the observed and modeled streamflow are each ranked by magnitude across the entire study period. After calculating ranks, however, we subset the data into periods with observed droughts at the various threshold levels of interest; for example, we took the lowest 20% of flow values for the 20th percentile drought threshold. Spearman’s r is calculated on these subsets as follows:

$$\text{Spearman’s r}={\displaystyle \frac{cov\left({R\left(obsQ\right)}_{ob{s}_{drought}},{R\left(simQ\right)}_{ob{s}_{drought}}\right)}{sigma\left({R\left(obsQ\right)}_{ob{s}_{drought}}\right)\ast sigma\left({R\left(simQ\right)}_{ob{s}_{drought}}\right)}}$$

(2)

where R(obsQ) and R(simQ) are the ranks of observed and modeled discharges, which are then subset into periods of observed drought. When the streamflow was at zero, we used the continuous dry period method, as described in Section 2.2, to break ties for ranking.

To investigate whether the model over or underestimated the total streamflow volume during drought periods, we calculated the percent bias of the observed flows below the drought threshold versus the modeled flows below the drought threshold ([24]; note that these are thresholds of the equal streamflow percentile, not of the streamflow volume). Our implementation of percent bias for drought flows focused only on flows below a threshold, like the implementation in Yilmaz et al. [74], although they used a 30th percentile threshold while we address a range of thresholds. Percent bias is calculated as follows:

$$PBias=100\ast {\displaystyle \frac{mean\left({simQ}_{si{m}_{drought}}\right)-mean\left({obsQ}_{ob{s}_{drought}}\right)}{mean\left({obsQ}_{ob{s}_{drought}}\right)}}$$

(3)

To provide a first-order estimate of errors in the statistical distribution of simulated flow magnitudes during periods of drought, we calculated the ratio of standard deviations between the modeled and observed streamflow (rSD) [24],

$$\mathrm{rSD}={\displaystyle \frac{{\sigma }_{{sim}_{drought}}}{{\sigma }_{ob{s}_{drought}}}}$$

(4)

This metric shows the relative variability between simulation and observations [24,47,75] and indicates if the model has over- or under-simulated the variability during periods of drought. The combination of rSD and percent bias evaluates whether the distribution of the simulated drought flows matches the distribution of the observed drought flows. Note that in the scorecard summaries, we present the absolute percent bias and the difference in the rSD from 1 so that both metrics can be presented in a range from better to worse.

2.3.3. Drought Signatures Evaluation

Duration, intensity, and severity are fundamental characteristics of drought events [76,77,78] and are widely used throughout the literature covering drought. The duration, intensity, and severity of a drought influence the impacts that the event has on socioeconomic and ecological systems [79]. It is important to evaluate how well models simulate these three drought characteristics as they are so heavily used by the research community and play a large role in the impact of droughts. To capture the event characteristics throughout all periods of interest, we aggregated the drought events identified each year to determine the annual signatures of drought in each given climate year. We built on a previous methodology dedicated to hydrologic signatures [80,81] to understand the signatures and characteristics of droughts. The drought signatures we present are duration, severity, and intensity, which capture a range of the important characteristics of drought [35]. Annual resolution signatures provide a continuous and matched time series between simulated and observed data. We then calculated the normalized mean absolute error (NMAE) on this annual time series to compare across stream gages and models:

$$\mathrm{NMAE}={\displaystyle \frac{ {\sum }_{i=1}^n|{sim}_i-{obs}_i| }{n}}\times {\displaystyle \frac{1}{mean\left(obs\right)}}$$

(5)

where n is the number of years, and sim_i and obs_i are the simulated and observed drought signature values (duration, severity, or intensity), respectively, for year i. Evaluating several different drought signature attributes is important as different indicators capture different information [35,82].

3. Results

We present our systematic model evaluation results for each metric category in three sections: event classification (Section 3.1), error components (Section 3.2), and drought signatures (Section 3.3). Figure 2 presents a scorecard with an overview of the model performance across all metrics for each model, threshold, and drought characterization method, with more detailed results presented in the following sections. Detailed result data from this systematic evaluation are published for both the NWM [83] and the NHM [84], so that they may be used as benchmarks against which other hydrologic modeling applications can be compared.

3.1. Event Classification

The NWM and NHM simulated droughts had moderate agreement with the observed droughts (Cohen’s kappa 0.41–0.60) for the fixed and variable methods at the 20th and 30th thresholds, but had only fair agreement (0.21–0.40) when simulating more severe droughts (5th and 10th percentiles; Figure 2). This difference highlights that examining both the fixed and variable methods over a range of thresholds is necessary in relaying the range and patterns of performance. The NWM had slightly but consistently higher Cohen’s kappa values than the NHM across both methods (fixed and variable) and all thresholds of characterizing drought (Figure 2). Additionally, the difference in the Cohen’s kappa between thresholds (for both the fixed and variable methods) is substantially larger than the difference between the modeling applications, with the NWM only showing the slightly better detection of drought events than the NHM, but there is a two-fold increase in the median between the most extreme threshold (5th percentile) and most moderate threshold (30th percentile) for each model and each method. Both modeling applications classify drought events similarly for fixed and variable drought methods, although the models have slightly more agreement between stream gages for the fixed drought methods (Figure S2).

The examination of model performance over varying hydroclimatic regions is needed to understand if hydrologic models can generalize and classify streamflow drought well across large national scales. We found that the overall streamflow drought classification performance of the NHM and the NWM varies by region, with regions in the wetter, eastern CONUS typically showing the better classification of drought events than regions in the drier, western CONUS (Figure 3). This is consistent across percentile thresholds and drought classification methods. One exception to the generally poorer drought event classification in the west is the wetter, most northwestern region of CONUS (region 12). The northwest region has a Cohen’s kappa that is slightly higher than the national median for both the fixed and variable methods for the NHM and generally for the NWM, except for the 5th and 10th percentile fixed thresholds and the 30th percentile variable threshold.

There were some minor differences between the modeling applications in the west and east of CONUS, with the NHM classifying drought occurrence slightly more accurately than the NWM in the Northwest, California, and Interior West for all threshold types. The NWM classifies drought occurrence more accurately at the variable 20th percentile threshold than the NHM for much of central and eastern CONUS, but the NHM performs better in the Northwest and California and Interior West regions.

There was also more regional variability for the fixed threshold than the variable-threshold classification (Figure 3 and Figure S3). The NWM also outperforms the NHM in eastern regions by a wider margin for the fixed method. The median values for each approach were similar across all thresholds, with a Cohen’s kappa between 0.41 and 0.47 for the 20th percentile threshold.

To further distinguish the model performance beyond regions, we examine the model performance across various basin characteristics like aridity, the drainage area, and the baseflow index (Figure 4; the baseflow index (BFI) is the long-term fractional contribution of subsurface flow to streamflow). Both modeling applications had a poorer performance with regards to classifying drought in more arid watersheds using the fixed and variable-threshold methods (Figure 4d and Figure S4). We found that both the NHM and NWM showed the worst simulations of drought occurrence for the 20% of stream gages with the highest BFI values (similar to [85,86]) and struggled for the 20% of stream gages with the lowest BFI (Figure 4e). The NHM and NWM also classified drought occurrence more accurately for moderate-sized basins (drainage area greater than 127.9 km² and less than 3238 km²) than for the largest and smallest basins (Figure 4f).

3.2. Error Components

Across all thresholds, the Spearman’s r of the NWM was higher than that for the NHM, indicating that the NWM more accurately simulates the timing of streamflows during drought (Figure 2, Figures S5 and S6). Like drought classification, the differences in Spearman’s r between thresholds were larger than the differences between models for the fixed method (Figure 2). The timing of streamflows during drought was also more accurately simulated in wetter than dry regions. The NWM performs better in most regions, especially in the Northeast and Northern Mid-Atlantic regions, although the NHM does better in three out of the four westernmost regions (Northwest, Rocky Mountains, California and Interior West) for fixed-threshold drought (Figure S5). Both the NHM and NWM captured some of the timing of flows during the observed droughts, but not all, with overall moderate correlations for the variable threshold across percentile thresholds (NWM, 0.58–0.69; NHM 0.48–0.58; Figure 2; [87]). Performance for the fixed threshold was worse for both modeling applications, especially the more severe droughts (e.g., 5th percentile) where both models had relatively weak correlations with the observed flows during drought (NWM, 0.26; NHM 0.24).

Overall, the NWM had greater absolute bias than the NHM across all thresholds, indicating that the NHM more accurately simulates the streamflow volume during drought (Figure 2). Both modeling applications overestimate the streamflow during fixed drought (20th percentile threshold median percent bias across stream gages of 9.4% NHM, and 34% NWM), which is in line with previous studies indicating that models commonly overestimate low flows [88,89,90]. Both modeling applications have lower median percent biases for streamflow during variable drought, although this bias was much higher for the NWM (20th percentile threshold median percent bias of −3.3% NHM and 22% NWM; Equation (3)). Consistent with the lower median bias, the bias in the NHM was also more balanced across sites: many sites over or underestimate drought flows, whereas the NWM overestimates drought flows for most sites, particularly in the central CONUS (Figure 5). Between modeling applications, the differences in bias were larger for more extreme droughts (5th and 10th percentile thresholds) than for modest droughts (20th and 30th percentile thresholds) (Figure 2). Additionally, the biases for individual stream gages were not strongly correlated between modeling applications (correlation for fixed drought = 0.22, correlation for variable drought = 0.25). The percent bias varies substantially by region, with smaller absolute biases in the wetter Northeastern and Northwest regions, but larger positive biases for dry western regions (No. 6, 9, 10, and 11), especially for the NWM (Figure 5c and Figure S7c). While the median percent biases were well balanced across the CONUS for moderate droughts for the NHM and moderately overestimated flow for the NWM, it is important to note that the median absolute percent biases ranged from 39.5% for the 30th percentile NHM to 79% for the 5th percentile NWM (Figure 2) and that for certain stream gages, particularly in southwestern regions, the percent biases exceeded 400% (Figure 5).

Figure 4. Cumulative distributions of model performance at stream gages for Cohen’s kappa exploring model performance under a variety of different conditions and model groupings. Sub figures plot Cohen’s kappa comparing (a) the performance between the National Water Model (NWM) and the National Hydrologic Model (NHM) with fixed and variable methods, (b) the performance between the NWM and the NHM at 5th, 10th, 20th, and 30th percentile thresholds, (c) the performance at reference vs. non-reference [91] and Hydro-Climatic Data Network (HCDN) vs. non-HCDN gages, (d) the performance for various quantiles of aridity (Text S2), (e) the performance for various quantiles of the baseflow index (BFI; Text S2), and (f) the performance for various quantiles of drainage area (DA; Text S2). In cases where it is not explicit, the model performance is shown for the fixed method at the 20th percentile threshold combining both NWM and NHM evaluations.

Figure 4. Cumulative distributions of model performance at stream gages for Cohen’s kappa exploring model performance under a variety of different conditions and model groupings. Sub figures plot Cohen’s kappa comparing (a) the performance between the National Water Model (NWM) and the National Hydrologic Model (NHM) with fixed and variable methods, (b) the performance between the NWM and the NHM at 5th, 10th, 20th, and 30th percentile thresholds, (c) the performance at reference vs. non-reference [91] and Hydro-Climatic Data Network (HCDN) vs. non-HCDN gages, (d) the performance for various quantiles of aridity (Text S2), (e) the performance for various quantiles of the baseflow index (BFI; Text S2), and (f) the performance for various quantiles of drainage area (DA; Text S2). In cases where it is not explicit, the model performance is shown for the fixed method at the 20th percentile threshold combining both NWM and NHM evaluations.

The difference in the ratio of standard deviations from its target value of 1 is higher for the NHM than for the NWM across all drought thresholds (Figure 2). There is relatively little agreement between each model’s ratio of standard deviations across stream g ages (fixed drought Spearman’s r = 0.27; variable drought Spearman’s r = 0.30). The spatial distributions of the ratio of standard deviations across the CONUS are variable (Figures S8 and S9). We generally see the lowest ratio of standard deviation differences between the modeling applications and the lowest variability among stream gages in the central eastern CONUS. The West, other than the Northwest, has high variability in the ratios of standard deviations across stream gages. The NWM tends to overestimate the variability of drought flows in these regions, while the NHM median ratio of standard deviations is close to 1. Both modeling applications tend to underestimate the variability in the Northeast.

3.3. Drought Signatures

The NHM and the NWM perform similarly when simulating drought signatures, as measured by the NMAE values of the time sequence of the annual drought duration, intensity, and severity values, compared to the variability between different signature types and between different thresholds (Figure 2 and Figure 6). Generally, both modeling applications have lower errors for drought duration and intensity, and higher errors for drought severity (Figure 2).

Both the NHM and NWM simulate lower drought duration errors using fixed-threshold methods than variable-threshold methods (Figure 2). Despite an overall difference of less than 5% in the NMAE values for drought duration, the correlation between models at individual stream gages is only moderate (fixed drought correlation = 0.81; variable drought correlation = 0.76). Both modeling applications were better at simulating the drought duration in the eastern and northwestern CONUS than in other regions. The better model also varies from region to region. The NWM performs better than the NHM in the Central Plains, but worse in the Southwest and California and Interior West.

Across all percentile thresholds, both the NHM and NWM have higher errors for drought severity than for drought duration and drought intensity, especially in the California and Interior West, Rocky Mountains, and Northern central plains (Figures S10 and S11). The NWM better simulates drought severity using the fixed drought method, while the NHM better simulates drought severity using the variable drought method. Both the NHM and NWM perform better at the 20th percentile threshold than at lower thresholds. Agreement between the NWM and the NHM at individual stream gages is lower for severity than for other drought signatures (fixed drought correlation = 0.56; variable drought correlation = 0.64), likely because drought severity is a measure integrating drought duration and intensity and because our drought duration and intensity signatures are standardized (unitless), unlike our severity signature of flow volume, which is inherently correlated with basin size and flow. Model performance is better in the eastern CONUS than in the western CONUS, with many stream gages in the western CONUS having NMAEs much greater than 1 (Figure 6). When simulating drought severity, the NWM tends to outperform the NHM in wet regions (1, 2, 3, 4, and 12) while the NHM performs better in drier regions (5–11).

Simulations of drought intensity are generally better than those of drought duration. The NHM better simulates drought intensity for both the variable and the fixed 20th percentile thresholds. The NWM, however, better simulates the 5th and 10th percentile fixed-threshold drought intensity. The agreement between the NWM and NHM performance for individual stream gages is greater for variable drought intensity than it is for drought duration (fixed drought correlation = 0.80; variable drought correlation = 0.84). Similar patterns emerge with regard to regional model performance, with a better ability to simulate drought intensity in the Northeast and Northern Mid-Atlantic regions and a poor ability in the western CONUS, apart from the coastal Northwest. The variability in performance is much less for drought intensity than it is for drought severity. With the 20th percentile variable-threshold drought method, the NHM performs better or equal to the NWM in nearly all regions.

Overall, both modeling applications struggle to simulate drought signatures, with a NMAE greater than 50% for all signatures except drought duration at the 30th percentile threshold and drought intensity at the 30th and 20th percentile thresholds. Both modeling applications show high error when simulating drought severity, with NMAE values ranging from 0.76 for the NHM at the 30th percentile variable threshold to 1.43 at the NHM 5th percentile variable threshold.

4. Discussion

4.1. Tradeoffs in Specific Model Performance

Each model has distinct tradeoffs when it comes to simulating aspects of drought. The “better” model depends heavily on the defined drought threshold and the method for determining drought severity (fixed or variable). We generally found that the NWM better reproduces drought occurrence, as well as the timing and variability of flow during droughts (as measured by Cohen’s kappa, Spearman’s r, and the ratio of the standard deviations, respectively), whereas the NHM better reproduces drought intensity signatures (for moderate and variable drought). Though our findings are consistent with those of Towler et al. [24], the additional streamflow drought metrics used in this study are needed to provide deeper insight into the models’ performance during hydroclimatic drought that the low-flow percent bias metric in Yilmaz et al. [74] cannot provide alone.

Tradeoffs in performance for different streamflow error components like magnitude versus timing between hydrologic models are commonly identified in the literature, often with results similar to our findings. Gudmundsson et al. [92] similarly found distinct tradeoffs between models’ mean and correlation errors (like metrics of magnitude and timing we use) when simulating European annual runoff cycles. Similar types of tradeoffs extend to simulations of drought. Some of these differences may be due to different initial development priorities for each modeling application (see Towler et al. [24], for modeling application descriptions). For example, the development of the NWM has focused on flood prediction and operates at an hourly timestep [93,94,95]. This may be the reason why the NWM better simulates the timing of drought events than the NHM. Another consideration is that the NHM was designed for water availability assessments, with a focus on simulating water quantity or magnitude, and operates on a daily timestep [96]. The calibration of the NHM included a step for matching the water balance volumes within each modeling unit, another step with the identified headwater catchments being calibrated using several non-streamflow datasets and a statistically generated streamflow to target the timing of streamflow; finally, the observed streamflow is used within each headwater catchment as a final calibration step [43]. Spatial frameworks are often different between models, where the NWM runs on a 1 km grid, while the NHM is based on hydrologic response units. These differences may have impacts on model performance. The calibration focus on water balance may be the reason why the NHM is better able to simulate streamflow volumes during drought rather than the timing. In the broader perspective, it is important to recognize each model’s original purpose when interpreting the comparison results using these streamflow drought metrics, and generally any metrics used in benchmarking.

4.2. Model Performance Exhibits Regional Variation with Better Performance in Wetter Eastern Regions than in Drier Western Regions

The NWM and the NHM both simulate drought more poorly in drier western regions of the CONUS than in wetter eastern regions. Previous studies have also indicated poorer overall and drought-specific model simulations in the western CONUS [24,48,58]. Climate factors [92], including precipitation and aridity [58,97], have been widely found to influence model performance in different regions, so the poorer simulations of drought in drier regions are not surprising. These findings highlight an important area for model improvement since arid regions of the western CONUS are often the regions where drought is a major concern.

We also note that several important factors that influence the water cycle in western CONUS are not well captured by the hydrologic models used in this study. For example, human water use, diversions, and reservoir regulations, which are common throughout western watersheds [98], are not fully represented in either model. Additional processes that are not thoroughly represented, like lake and stream channel evaporation [99] or deep and complex groundwater systems [100], also may impact the model performance in the western CONUS. Other studies have shown that missing groundwater processes and not accounting for human modifications can result in poor model performance [73,101]. Benchmarking both models across our suite of metrics may help guide future development priorities in each model when looking to address issues such as the representation of human influences. If adding reservoir representation to the models, the NHM might benefit from more focus on the improved representation of the impacts of reservoirs on flow regime timing, as it had a poorer performance in simulating flow timing metrics (Spearman’s r) during drought relative to the NWM. The NWM, on the other hand, could potentially benefit from more focus on the improved representation of the flow magnitude impacts of reservoirs, as it had a poorer performance when simulating the flow magnitude (percent bias) during drought relative to the NHM.

The relative performance also varies spatially. For example, the NHM simulates drought occurrence (as measured by Cohen’s kappa) better in the western coastal regions (11 and 12) despite a worse national performance. In the southcentral CONUS, both modeling applications overestimate the magnitude and variability of drought flows by a large margin (like findings across all flows by Towler et al. [24]). Both modeling applications, especially the NWM, tend to better simulate the timing of flows during drought (Spearman’s r) in the southcentral CONUS, with values close to the national median. The Rocky Mountain region is the reverse: both modeling applications simulate the magnitude and variability of flows during drought similarly well to how they do in other regions, but Spearman’s r for both models is the lowest of all regions, which indicates difficulties simulating the timing of flows during drought in the Rocky Mountains. The regional differences in which the components of error each model perform well or poorly highlight some of the difficulties and potential tradeoffs between these models. Efforts focusing on improving the simulation of the magnitude of flows during drought may improve drought simulations for the southcentral CONUS, where the model has relatively high biases during periods of drought. In contrast, the Rocky Mountains, where the model simulations have a low Spearman’s r with observations, might benefit most from efforts focused on improving the simulations of flow timing during drought.

We see similar tradeoffs between metrics in regions where both modeling applications perform relatively better. The NWM and NHM simulate the timing, magnitude, and variability of flows during drought well in the northeastern CONUS with median statistical values that are either the highest or close to the highest compared to other regions. The NHM performs less well when simulating the occurrence of fixed-threshold drought in the northeast relative to some other regions like the Northern and Southern Mid-Atlantic (although Cohen’s kappa values are still at the national median). This indicates that the model’s simulations during periods of drought are good, but that the NHM may not always correctly indicate that there is a drought relative to the Mid-Atlantic regions, where the NHM better simulates drought occurrence.

4.3. Larger Differences in Performance Between Thresholds than Between Models Highlights the Importance of Using Consistent Benchmarking Methods

The differences in Cohen’s kappa, Spearman’s r, the drought duration, and drought severity was greater between thresholds than between modeling applications. Both the NHM and NWM performed worse for more extreme droughts. This may in part reflect similar deficiencies in the climate input data, the applicability of the model equations to arid regions, or the model parameterizations [24]. Once calibrated, models typically have less variance than the original observations, so they tend to overpredict low flows and underpredict high flows [102]. However, we did not see more modeled than observed variability within these drought periods, as measured by the ratio of standard deviations. Both modeling applications may be able to simulate the impact of flow generation processes on moderate drought flows (e.g., 20 and 30%) but have difficulties simulating the processes that drive extreme drought flows (e.g., 5 and 10%). Additionally, these severe flows are hard to calibrate as they occur infrequently.

4.4. Summary

The results show a varied performance between the modeling applications that differs based on the metric being evaluated, the method of identifying drought, and the region or stream gage of interest. These results suggest that these models can provide benefits and useful information in certain contexts and regions, particularly for more moderate droughts in high-performance regions like the eastern CONUS. Users of these modeling applications should exert caution because some evaluation metrics indicate potential issues. For example, some stream gages have percent biases over 400% during periods of drought and some have high NMAE values when simulating drought severity. These factors suggest that the model may be problematic for certain applications, particularly in capturing more severe droughts in certain regions of CONUS such as the drier western regions.

The systematic evaluation of these two modeling applications highlights differences in performance by metric and region. These differences, along with the larger differences between drought thresholds than between modeling applications for many performance statistics, show the importance of the systematic benchmarking of different models and the use of the same methods of drought identification when comparing models. It also shows the importance of evaluating a range of different thresholds, in a range of different regions, using various performance metrics.

The consistent finding that both modeling applications have difficulties in simulating more extreme drought and drought in more arid regions highlights the limitations of both models and suggests a need for continued modeling improvement, as also indicated by others [26,73]. The improved simulation of severe drought flows and drought in arid regions is critical as these flows have large social and ecological impacts. This need for improved simulation may become increasingly important as, in some of these arid regions, extreme drought flows may become more common with a changing climate [103]. It may be especially important for regions such as the Colorado River Basin that experience major droughts with substantial impacts across social and ecological systems in the region [104,105].

4.5. Limitations

This study helps improve the understanding of model performance in simulating drought events, but there are important limitations to note. Our assessment does not control for differences in the model calibration or forcing datasets, which influence the simulated results. For ideal comparisons of the underlying models, both modeling applications would use the same forcings and calibration techniques; however, this can be impracticable for large simulations as they are typically funded by different entities and created for somewhat different purposes. Similarly, the evaluation is performed on simulations at stream gages used for calibration and therefore does not assess out-sample errors, but it is likewise usually impractical in large-domain model implementations to perform the simulations needed to address this issue. Nevertheless, benchmarking the specific implementations of modeling applications or built-for-comparison modeling applications still gives insight into how each can be used for research and improvement. Our study assesses both fixed and variable methods of characterizing drought at four thresholds, but there are many other methods of characterizing streamflow drought that could yield different evaluation results. The autocorrelation of daily values is not considered in our study design, though this is critical to address if planning to perform statistical significance testing between modeling applications. Uncertainty in hydrologic simulations stems from several sources, including uncertainties in the input forcings, as well as from the hydrological model structure, process representation, absence of anthropogenic processes, and parameterization. Although we do not explicitly investigate model uncertainty nor their interactions here, we do note that we use two different hydrological models (NHM and NWM), which could be conducive to developing a multi-model ensemble; studies have shown that the ensemble mean can outperform individual models [106].

5. Conclusions

This study presents a comprehensive approach to evaluating simulations of streamflow drought and applies it to two conterminous U.S.-scale hydrologic modeling applications, the National Water Model (NWM) and the National Hydrologic Model (NHM). Our comparisons between the NWM and the NHM show varied results. The NWM tends to better simulate the timing of streamflow during drought events (measured by Spearman’s r) while the NHM tends to better simulate the magnitude of flow during drought events (measured by percent bias). There were also strong spatial trends in the drought simulation performance, with both the NHM and NWM performing better in wetter regions than drier ones, creating a stark east versus west divide in performance. Thus, both modeling applications perform worse in drought simulations for the regions that are most susceptible to drought. Finally, the differences in performance were typically greater between different drought thresholds than between either modeling application, with both the NHM and NWM exhibiting difficulties in simulating the most severe drought events.