Optimizing Analog Ensembles for Sub-Daily Precipitation Forecasts

Abstract

This study systematically explores existing and new optimization techniques for analog ensemble (AnEn) post-processing of hourly to daily precipitation forecasts over the complex terrain of southwest British Columbia, Canada. An AnEn bias-corrects a target model forecast by searching for past dates with similar model forecasts (i.e., analogs), and using the verifying observations as ensemble members. The weather variables (i.e., predictors) that select the best past analogs vary among stations and seasons. First, different predictor selection techniques are evaluated and we propose an adjustment in the forward selection procedure that considerably improves computational efficiency while preserving optimization skill. Second, temporal trends of predictors are used to further enhance predictive skill, especially at shorter accumulation windows and longer forecast horizons. Finally, this study introduces a modification in the analog search that allows for selection of analogs within a time window surrounding the target lead time. These supplemental lead times effectively expand the training sample size, which significantly improves all performance metrics—even more than the predictor weighting and temporal-trend optimization steps combined. This study optimizes AnEns for moderate precipitation intensities but also shows good performance for the ensemble median and heavier precipitation rates. Precipitation is most challenging to predict at finer temporal resolutions and longer lead times, yet those forecasts see the largest enhancement in predictive skill from AnEn post-processing. This study shows that optimization of AnEn post-processing, including new techniques developed herein, can significantly improve computational efficiency and forecast performance.

1. Introduction

Numerical weather prediction (NWP) models are impaired by imperfect initial conditions and simplified approximations of physical concepts. Statistical post-processing can improve forecast quality by reducing systematic model errors [1]. Common bias-correction procedures include regression methods [2,3,4], model output statistics [1,5,6,7,8], Kalman-filtering [9], moving- and weighted-average techniques [10], Bayesian model averaging [11,12], machine learning [13,14,15], and analog ensembles (AnEns; [16,17]); many of these categories have overlapping techniques. Previous research has demonstrated successful applications of AnEns to temperature [17,18], wind speed [17,19,20,21,22,23], wind power [24,25,26], solar radiation [24,27], air quality [28,29,30], precipitation [16,31,32,33], and streamflow predictions [34].

Originally, analog methods were based on the assumption that atmospheric conditions tend to follow recurring patterns and hence could estimate future weather from similar past developments. More successful recent versions of the technique operate on the assumption that past similar conditions will have similar model errors. In an operational framework, the AnEn technique post-processes a target model forecast by searching for similar (i.e., “analog”) model forecasts in the past and using their corresponding observations to compose an ensemble [9,16,17,18]. Since the AnEn samples the observed distribution without assuming a target distribution, this nonparametric method directly corrects systematic model error. The analog procedure constructs probabilistic ensemble forecasts from a history of raw deterministic model forecasts from only one NWP model run. Hence, they require a fraction of the computational expense of a traditional multi-model or multi-run NWP ensemble.

The success of AnEns has two main requirements: (a) the availability of a consistent high-quality meteorological archive of model forecasts and concurrent observations—the longer the archive the better; and (b) a definition of “similarity”, which measures the degree of analogy of past forecasts with a target forecast to identify a set of best analogs that construct the AnEn.

Similarity is assessed primarily using predictors, for instance via a multivariate Euclidean-distance measure. Usually predictor variables have varying degrees of importance (for example, related to location and season [35]) and can be weighted accordingly. Some studies neglect these dependencies and utilize equal [9,17,18,36,37] or arbitrary weights [16,22,38] for a subjectively reasonable selection of predictors. Other studies use domain knowledge to design “levels of analogy” [32,39,40,41,42,43], where sequential selection levels sort successive subsets of analog candidates from the meteorological archive. More recent studies derive predictor weights objectively from correlation coefficients with the predictand [30], or obtain them directly from a predictor selection procedure, such as brute force (BF; [35]), forward selection (FS; [44]), principal component analysis (PCA; [24,26,45,46]), or genetic algorithms [42,47].

BF is a popular optimization approach as it identifies optimal weights by testing all possible combinations. However, the computational expense of running and evaluating AnEns in numerous variations becomes infeasible for a large number of predictors. Therefore many studies [14,21,33,48] pre-select a reduced number of predictor candidates before BF optimization. The pre-selection is sometimes based on domain knowledge [19,21] or an efficient filter method (such as correlation analysis; [33]), however it may limit the BF predictor optimization by unintentionally disregarding useful predictors to begin with. References [44,49] developed a more efficient BF approach, the stepwise FS (also used in [33,50]), which iteratively tests the AnEn performance by adding any of the predictor candidates to the previously selected predictors one step at a time. This approach resembles BF results closely [33], while the computational savings enable the exploration of more predictor variables. However, compared to filter methods, the FS approach still requires significant time and resources to train and analyse the predictor optimization. Despite the known advantage of optimized predictor weights [35], the computational effort continues to represent an obstacle.

The similarity measure for the analog search can also account for the temporal trend of the predictors across sequential forecast lead times [17]. This consideration has been shown to improve AnEns [9]. A few studies [9,17,25] investigate the impact of the time window length, while others use an arbitrary window length of +/−1 forecast steps [21,29,35,44,50], which can result in different total window widths depending on the forecast interval. Moreover, different physical variables can have different autocorrelation characteristics and hence the ideal time window to match the temporal predictor trends may vary accordingly.

Real-time AnEns require operational NWP with a long and consistent data history. Reforecast datasets therefore have a great potential for data-driven methods like AnEns [38,51], especially if the same model configuration is tuned for local characteristics and is still used operationally in real time. However, many studies [31,32,33,43,52] train the AnEn on long reanalysis datasets targeting the development of statistically downscaled data products, or serve as proof of concept for an operational framework (i.e., perfect prognosis). References [43,53] showed that the quality of reanalyses affects the AnEn skill significantly—sometimes even more than the choice of predictors. Since reanalyses includes more data assimilation of atmospheric measurements, operational forecasts have faster (and possibly different) error growth and NWP quality is likely to have a similar or larger impact on AnEns than the reanalyses.

Compared to other meteorological variables, precipitation has high spatial and temporal variability, making post-processing particularly challenging. Precipitation distributions are also skewed towards zero and can be represented with a discrete distribution regarding event thresholds (e.g., rain vs. no-rain), or a continuous distribution when considering quantitative amounts.

The decreasing number of precipitation events with increasing intensity represents a statistical disadvantage for post-processing high-impact events with data-driven methods, such as AnEns. A long data history is required to ensure sufficient sampling when searching for analogs of relatively rare events. For the purpose of expanding the analog search data pool, ref. [38] introduced the concept of supplemental locations, which uses additional grid points or stations with similar climatology and terrain characteristics. This technique significantly improves heavy precipitation forecast calibration [38] but requires a relatively large domain with numerous stations. Instead of expanding the sample size using spatial supplements, ref. [54] suggested a moving-time-window approach to inflate the meteorological archive using temporal supplements. When searching for analogs for a daily-precipitation target, ref. [54] considered not only the same lead times, but also 24-h totals that result from sub-daily offsets to the target lead time.

Most precipitation AnEn studies investigate daily accumulation totals [16,32,40,42,43,52,55,56,57], which exhibit better predictability than sub-daily amounts [58]. However, resolving the sub-daily variability of precipitation has value to many weather forecast applications, such as flood management, infrastructure maintenance (e.g., transportation and construction), agriculture (e.g., soil erosion and crop damage), and smaller hydroelectric operations where flows are managed at sub-daily time steps. Reference [59] temporally disaggregated daily precipitation analog forecasts to hourly time steps. Other AnEn studies directly derive 12-hourly [38,60] or 6-hourly precipitation amounts [33,36,41,61,62], whereas NWP forecasts are commonly considered at hourly intervals.

Our study domain is southwest British Columbia (BC), Canada. BC’s complex terrain amplifies a variety of forecasting challenges, such as imperfect numerical and physical approximations in NWP and the high spatial variability of the surface conditions. For example, the prediction of orographic precipitation enhancement on windward slopes and lee-side rain-shadow requires adequate representation of topography (e.g., slope steepness), initial state (e.g., upstream conditions over the Pacific Ocean), dynamics (e.g., flow and stability), and physics (e.g., mixed-phase microphysical processes). Inaccuracies in any NWP components cumulatively contribute to model errors and make post-processing a crucial factor in improving precipitation forecast quality over regions of complex terrain such as BC. Southwest BC sees copious precipitation during the cool season, which, among other impacts, has led to catastrophic flooding events [63,64,65]. Additionally, skillful precipitation forecasts are crucial because hydropower contributes the bulk of the total electricity production in BC, and precipitation forecasts are used to plan generation system operations and mitigate flood risk.

This study demonstrates the optimization of AnEn forecasts for sub-daily precipitation in southwest BC by post-processing one of the regionally best performing configurations following [58]. As described in Section 2, we apply the AnEn method as a station-based post-processing tool to statistically downscale the deterministic model forecasts—an approach that is suitable for real-time operational model post-processing. This paper builds on existing methods to optimize AnEn parameters and explores new variations in the AnEn methodology that either improve forecast performance or computational efficiency.

In Section 3, we first compare predictor selection techniques and suggest a more efficient FS approach. Next, we investigate the impact of the temporal trend similarity, while assessing accumulations from daily to hourly intervals. As such, to our knowledge, this is the first paper demonstrating successful AnEns for hourly precipitation forecasts in this form. Finally, we redesign [54]’s moving-time-window approach for shorter accumulation windows to make the best use of a limited meteorological archive in finding the best available analogs. Our verification shows the improvement in each optimization step and reveals the trade-off between temporal resolution and precipitation predictive skill following AnEn post-processing. Section 4 gives the summary and conclusions.

2. Materials and Methods

2.1. Data

The NWP data for this study are from the Weather Research and Forecasting (WRF) model [66] version 3.8.1 with the Advanced Research WRF (ARW) dynamical core, initialized with the Global Deterministic Prediction System (GDPS) model [67,68] from Environment and Climate Change Canada (ECCC). The model setup, including initialization, domains, and physics, was chosen based on [58]—a study that evaluated precipitation forecasts from over 100 systematically varied model configurations. We use the WRF configuration from that study that performed above average across verification scores and performed best for 75th-percentile (75p) equitable threat score (ETS) and probability of detection (POD)—namely, the WRF single-moment 5-class microphysics scheme (WSM5; [69]), the Kain-Fritsch cumulus scheme (KF; [70]), the Yonsei University turbulence scheme (YSU; [71]), and the multiphysics Noah land surface model (Noah-MP; [72,73]).

Since [58] found smaller raw forecast errors for coarser grid spacings, but finer grid spacings are assumed to better resolve the spatial variability over complex terrain, this study focuses on the mid-size domain with $\Delta x=$ 9-km. WRF runs are initialized daily at 00 UTC and provide 3 days (72 h) of hourly forecast data after 9 h of spinup. Hence, each forecast day starts at 0900 UTC (0100 Local Standard Time). For further details on other WRF settings and the verification results, see [58].

Utilizing this regionally optimized WRF configuration, we generated a 5.75-year reforecast dataset from January 2016 through September 2021.

Table 1. Physical variables extracted from the WRF model output and considered as predictors.

Variable	Abbreviation	Levels
Total Precipitation	PCP	Surface
Integrated Water Vapor *	IWV	Column
Integrated Vapor Transport *	IVT	Column
Water Vapor Mixing Ratio	r	2 m, 70 kPa, 50 kPa
Specific Humidity *	SH	70 kPa, 50 kPa
Relative Humidity *	RH	70 kPa, 50 kPa
Moisture Index 1 * ($R{H}_{70kPa}\times IWV$)	MI1
Moisture Index 2 * ($R{H}_{70kPa}\times W_{70kPa}$)	MI2
Temperature	T	2 m, 70 kPa, 50 kPa
Potential Temperature	Th	2 m, 70 kPa, 50 kPa
Dewpoint Temperature	Td	70 kPa, 50 kPa
Total Totals Index *	TT
K-Index *	KI
U-component Wind	U	10 m, 70 kPa, 50 kPa
V-component Wind	V	10 m, 70 kPa, 50 kPa
W-component Wind	W	70 kPa, 50 kPa
Wind Direction *	WD	10 m, 70 kPa, 50 kPa
Wind Speed *	WS	10 m, 70 kPa, 50 kPa
Sea Level Pressure	SLP	Sea Level
Surface Pressure	SfcP	Surface
Geopotential Height	GPH	70 kPa, 50 kPa
Boundary Layer Height	PBLH

* Derived from WRF output variables.

lists 22 physical variables that were extracted or derived from the WRF output, some at different vertical levels, resulting in 41 variables total. The list includes general atmospheric parameters that characterize moisture, thermal, stability, and wind conditions. Some variables like MI1 and MI2 were inspired by other precipitation AnEn studies [43,52,54] that used these as predictors.

The model variable “PCP” includes precipitation in all forms (i.e., rainfall, snow, sleet, etc.) and is represented as liquid equivalent. However, BC’s South Coast has a mild climate year-around and freezing temperatures and snowfall are rarely observed at lower elevations.

The original temporal resolution of WRF output is hourly. When assessing longer accumulation windows in this study (i.e., 3-, 6-, 12-hourly, and daily intervals), the sum of hourly PCP is calculated, whereas all other model variables are estimated by their time average. This study considers discrete (non-overlapping) and rolling (overlapping) windows for the accumulated datasets. Discrete-window results are useful for comparison with other studies that use such windows. However, they can split precipitation events unfavorably, making them seem longer and weaker.

Rolling windows, on the other hand, sample the same precipitation events in hourly offsets and ensure the capture of maximum rates for any event and accumulation interval. Therefore they provide a more complete picture in assessing the impact of temporal resolution on predictability. The hourly time step of rolling windows can further benefit temporal trend similarities and supplemental lead times (described below), for which longer-accumulation discrete windows often have too large of a time step. However, it is important to remember that rolling windows possess overlap from one time step to another, which makes them temporally correlated.

46 stations from two networks (ECCC and BC Hydro) provide hourly precipitation observations within the domain of interest, shown in Figure 1. BC Hydro station observations are manually quality controlled at BC Hydro [74]. Additional quality control checks on both observational networks ensure that

• each station has at least 90% of data available, and missing data is not systematically distributed (e.g., in the same season, at the same time of day, or over a large consecutive period); and
• outliers are reasonable considering the synoptic situation (e.g., convection), nearby stations, and the available station climatology.

The gridded model data is spatially interpolated to the station locations using the nearest-neighbor approach. The matched station and model dataset is split into 4.75 years (January 2016 through September 2020) for training/optimization using a leave-one-out approach, and one year (the 2021 water year: October 2020 through September 2021) for independent testing/verification. The optimization and verification process both search for analogs from the same 4.75-year training dataset. However, to ensure data independence during optimization, the leave-one-out approach excludes a buffer of ±15 days surrounding the targeted initialization.

Since higher precipitation rates have larger impact and are more challenging to forecast, this study optimizes for performance on “moderate” or heavier precipitation intensities—specifically, 75th percentiles (75p). Section 3.4 provides additional verification results for 90th-percentile (90p) events, i.e., “heavy” precipitation rates. The thresholds are calculated at each station based on observations after excluding values <0.25 mm (which for many rain gauges is the smallest measurable amount). Percentile values vary with accumulation window; examples of frequency distributions are shown in Appendix A.

2.2. Analog Ensemble Methodology

Analog model forecasts (AnFcsts) are a set of past model forecasts (PaFcsts) that, in regard to selected variables and similarity metrics, are most similar to the target model forecast (TaFcst) at a given lead time and location. The past verifying station measurements that correspond to the AnFcsts—the analog observations (AnObs)—are used as ensemble members to compose the AnEn (see Figure 2). The AnEn is considered to be the post-processed version of the deterministic raw TaFcst, and hence should provide a better forecast for the verifying observation (VerifObs) at the target time. Thus, the analog selection is determined solely from the model space, whereas the AnEn is composed solely of samples from the observation space.

Although this study investigates univariate AnEn forecasts for precipitation (i.e., the predictand), a distance measure assesses the multivariate similarity between the TaFcst and all PaFcsts. The best ranking PaFcsts are chosen to be the AnFcsts and are determined independently for each station location and forecast lead time. We use a popular similarity metric developed by [9] that calculates similarity scores

$$\parallel {\mathrm{TaFcst}}_t,{\mathrm{PaFcsts}}_t\parallel =\sum _{v=1}^{N_v}\frac{w_v}{{\sigma }_{v,t}}\sqrt{\sum _{j=-\tau }^{\tau }{({\mathrm{TaFcst}}_{v,t+j}-{\mathrm{PaFcsts}}_{v,t+j})}^2},$$

(1)

for each PaFcst with the TaFcst at lead-time t relative to model initialization. $N_v$ is the number of physical variables v (i.e., predictors) for which closeness between AnFcsts and the TaFcst is desired. The variable weights $w_v$ can assign larger importance to variables with stronger predictor relationships. The division by each variable’s standard deviation ${\sigma }_{v,t}$ in the model training dataset at lead time t standardizes the variables and makes the similarity scores dimensionless. $\tau$ defines a lead-time window that centers the target lead time t and includes additional lead times over which the similarity in the temporal trends of the predictors is computed.

The physical variables that are used for the analog search should exhibit good predictor relationships with the predictand. Precipitation AnEn studies often use model PCP and IWV as predictors [14,16,38] with weights 0.7 and 0.3, respectively, [16,38]. We use these variables as reference when evaluating the predictors that we obtain from other predictor selection techniques (presented in Section 2.2.1). The control run further uses $\tau$ = 0, thus only matching the predictor values of TaFcst and PaFcsts at lead time t. The AnEn sensitivity to $\tau$ is investigated in Section 3.2. With these reference settings, the optimal AnEn size was found using approximately 30 AnFcst members on average (not shown), which agrees with [33], and is therefore used throughout this study.

To prioritize forecast performance for significant events that are more critical for decision makers, in this study the AnEn parameters are optimized using the 75p threshold-weighted continuous ranked probability score (twCRPS; [7], see Appendix B.1).

2.2.1. Predictor Selection Procedures

The predictor selection procedure objectively assesses which of the 41 model variables in Table 1 are the best predictors for precipitation. Initial investigation of predictor relationships between all variables and observed precipitation using filter methods (e.g., correlation analysis) reveals variability in predictor importance across stations and months (see Appendix C). This is expected since locations and seasons in southwest BC can exhibit different characteristics due to topography and climate. Therefore, we investigate optimal predictors and their weights independently at each station location and meteorological season.

A brute force (BF) approach [35] is a method that runs the AnEn optimization on all combinations of predictor weights (to a defined precision), and determines the best predictor combination according to an evaluation score. Since this method is computationally very expensive, [44] suggested a step-wise forward selection (FS) method, which sequentially selects one predictor at a time by BF testing all weighting options only among the step-wise selected predictors. The first step tests all $N_v$ variables as single predictors with $w_v=1$ (see Equation (1)), and the variable resulting in the best evaluation score is chosen as the first predictor. The next step selects the second predictor by testing the remaining $N_v-1$ variables in combination with the already selected first predictor. The weights applied to each variable are tested by BF, i.e., all possible combinations. For instance, using weight increments of 0.1 in the interval [0, 1] with the constraint that the sum of weights is always 1, results in 9 options for two variables. Selecting the third predictor has $(N_v-2)\ast 84$, and the fourth predictor has $(N_v-3)\ast 126$ options, etc. Predictors are selected if they improve the evaluation score by a chosen increment compared to the score in the previous FS step (e.g., mean absolute error of 3% in [44]). This way, different stations can receive different numbers of predictors. Although this FS approach is considerably faster than BF, the computational cost is still significant for large $N_v$ and high weight precision (i.e., smaller increments).

As a further computational reduction of [44]’s FS, we propose an “efficient FS” (EFS) as follows. Assuming that those variables first selected as predictors have larger importance, we constrain the weighting options to $w_{Predictor1}\ge w_{Predictor2}\ge ...\ge w_{Predictor{N}_v}$. This way the second predictor has $(N_v-1)\ast 5$, the third predictor has $(N_v-2)\ast 8$, the third predictor has $(N_v-3)\ast 9$, and the fourth predictor has only $(N_v-4)\ast 7$ options. We further define the first predictor to be PCP without testing. We proceed to optimize predictors for twCRPS until the improvement drops below 1%.

We investigate four variants of FS to determine predictor weights:

• All-EFS: Using the EFS to test all 40 variables in addition to PCP as predictors.
• DC-FS: Using [44]’s FS to test a subset of 10 variables as predictors. The 10 predictor candidates are pre-selected based on the highest distance correlation coefficient (DCorr; a measure that identifies both linear and non-linear relationships [75]) with observed precipitation.
• DC-EFS: Using the EFS to test the same subset of 10 predictor candidates as in DC-FS.
• DCV-EFS: Using the EFS to test a subset of 10 variables as predictors, except here, the predictor candidates are based on the best DCorr, as well as the variance inflation factor (VIF, a measure of multicollinearity among variables). Specifically, we grow a set of 10 predictor candidates by sequentially adding one variable at a time, starting from the best ranking DCorr, provided the VIF among the growing set of predictor candidates stays below a threshold value of 10. If this threshold is exceeded it means that the variable exhibits strong correlation with other variables that were already selected and we assume that this variable contributes no additional value as a predictor for the AnEn. Since some of our 41 variables are related (e.g., the same variables at different vertical levels), the VIF check limits the use of correlated and presumably redundant variables in the FS.

These experiments aim to investigate (a) whether the EFS is competitive with [44]’s FS, and (b) whether DCorr or DCorr and VIF can effectively pre-filter meaningful predictor candidates, reducing computational expense compared to testing all variables in EFS. For the feasibility of testing all these methods, we conducted this optimization of predictor weights only for 3-hourly discrete accumulation windows and day-1 forecasts.

We further conducted principal component analysis (PCA) on the standardized datasets of all variables and the 10-variable subsets resulting from DCorr and DCorr-VIF analyses. Different experiments with the principal components (PC) as predictors and their weights either using the eigenvalues or from EFS on the PCs, were not competitive with the (E)FS methods described above and are therefore not shown in this paper. Ref. [35] obtained similar results comparing BF and PCA predictor-selection methods.

2.2.2. The Supplemental-Lead-Time (SLT) Approach

The original AnEn approach, as described above, searches AnFcsts only across those lead times in the training period that match the target lead time of the TaFcst. The aim of matching lead times is to detect AnFcst candidates with error characteristics similar to the TaFcst. It further ensures that the AnFcsts are sampled from the same time of day, which is particularly important for predictands like temperature and wind that exhibit diurnal cycles. However, precipitation in southwest BC has no significant diurnal cycle in the cool-season when most precipitation occurs; and it is plausible that better AnFcst candidates are available at other lead times surrounding the target lead time, for which model errors are still similar.

Thus, we explore the use of “supplemental lead times” (SLTs) in Section 3.3. As exemplified in Figure 3, AnFcst candidates are considered over a range of offsets from the target lead time. Since AnFcsts from the same past model initialization would be temporally correlated, we only select the best single PaFcst among SLTs from each initialization. Hence, this method selects the AnFcst from a SLT (different from the target lead time) only when the score resulting from Equation (1) indicates closer similarity (i.e., a better analogy).

Accumulation-window treatment must be considered when using SLTs. For instance, ±2 SLTs applied to hourly forecasts selects the best out of five PaFcst options over a lead-time window width of five hours; ±2 SLTs applied to 3-hourly rolling forecasts considers five PaFcst options over a window width of eight hours; and ±2 SLTs applied to 3-hourly discrete forecasts considers five PaFcst options over a total window width of 15 h. Since SLT consideration for longer discrete windows quickly inflates the effective lead-time window width over which model error growth can be significant, we examine SLTs only for short and rolling accumulation windows.

This method differs from [54], who inspected 12-, 6-, and 3-hourly offsets of 24-h accumulations for daily precipitation targets by including all offsets within the 24-h period. For example, [54]’s best-performing 3-hourly offsets result in 8 times as many AnFcst candidates as the original AnEn approach, hereby artificially inflating the meteorological archive with temporally correlated PaFcsts. Our SLT approach, on the other hand, increases the number of AnFcst candidates indirectly by choosing only the single best candidate in the SLT window centering the target lead time, without risking the selection of multiple correlated PaFcsts.

3. Results and Discussion

3.1. Predictor Selection Optimization

The four iterative FS and EFS approaches described in Section 2.2.1 determine predictor weights from the training dataset only. Following optimization for 75p twCRPS, all methods require that the resulting predictors yield better or equal twCRPS compared to the control predictors over the training period. Because the set of 10 predictor candidates used in DC-EFS or DCV-EFS are each a subset of the variables used in All-EFS, it is also required that All-EFS yields better or equal twCRPS during training. It is further required that DC-FS is better than DC-EFS during training, since both methods are preconditioned by the same predictor candidates and the FS approach has more weighting options than EFS. However, running the AnEn over the testing period with the predictor weights determined from the training period reveals whether the selection procedure actually led to the required improvement or whether it overfitted the training dataset.

Figure 4 compares twCRPSS among the four methods, segregated by training and testing. Recall that predictor weights are optimized independently for each station and meteorological season. Therefore, Figure 4 summarizes the twCRPSS of all 46 stations in boxplots and panels for each season.

For all seasons the majority of stations see improvement in twCRPS following predictor optimization with any method. This is true for both training and testing, which indicates that all methods are beneficial. However, the consistent shift between training and testing scores suggests some overfitting; namely, the best predictors during training are still good—but potentially not the best—predictors during testing.

Since summer in BC is by far the driest season, differences in small values of twCRPS between control and optimized predictors are amplified in twCRPSS, hence the larger improvement on average and the large spread. Warm-season dry periods can extend into spring and fall months. The winter season contains the most consistent precipitation pattern.

All methods significantly (see Appendix B.2 for significance testing) improve twCRPS compared to the control during training and testing. However, the differences in twCRPSS among methods are relatively small. Compared to the training, the testing score distributions across stations are less often significantly different among methods. However, winter All-EFS scores are still significantly better than DC-FS and DCV-EFS during testing. Although not always significant, All-EFS is consistently best during training and continues to be best on average (not shown) during testing in summer, fall, and winter. Therefore, we use the predictors resulting from the All-EFS method (see Appendix D) hereinafter. DC-EFS and DCV-EFS are competitive with All-EFS but require considerably less optimization time by assessing only a quarter of the variables. Therefore, they would be viable alternatives for predictor optimization in other studies.

These results show that the EFS approach is capable of effectively reducing the computational effort of predictor tuning compared to [44]’s FS method, while maintaining similar (and sometimes even better) improvements compared to static control predictors.

3.2. Temporal Trend Similarity

In addition to predictor-variable choice and their relative weights, the temporal trends of predictors can also help to better identify AnFcsts. Temporal trend similarity (TTS) is assessed over a time window centered on the target lead time, and window width results from the definition of $\tau$ ranging over a number of time steps. The total window width depends on the accumulation window and steps. For instance, a TTS window using ±2 time steps (i.e., $\tau =2$ in Equation (1)) covers a 5-h period for hourly precipitation, or a 7-h period for 3-hourly rolling windows, whereas it covers an effective 15-h period for 3-hourly discrete windows.

We investigate the impact of TTS with $\tau$ ranging from 1 to 5 time steps on twCRPS relative to using no TTS ($\tau =0$). Since autocorrelation between the predictors and the predictand is unlikely to exceed half a day, for long discrete accumulation windows we discard TTS calculations that would result in effective window widths >36 h. Therefore, 24-hourly discrete windows are not considered for TTS, and 12-hourly discrete windows are assessed only for a TTS with $\tau =1$. Note also that there are no forecast values available preceding (following) the first (last) lead time in a forecast series from one initialization; therefore, a few lead times at the beginning and end of a forecast series do not experience the full effect of TTS.

Figure 5 and Figure 6 show that hourly and rolling windows, all of which have hourly time steps, benefit most from TTS. Discrete windows with longer accumulations, and hence larger time steps, cover longer total lead-time widths over which TTS is often less effective—here only on days 2 and 3 is twCRPSS sometimes positive (better) for $\tau >0$.

Longer forecast horizons generally obtain better twCRPSS for longer time windows. Due to error growth with lead time, day-1 forecasts exhibit better predictability than day-3 forecasts and thus, day-3 forecasts have greater total potential for improvement. At longer forecast horizons it appears that the added temporal dimension in the similarity consideration somewhat balances the abating quality of the predictor variables, whereas at shorter forecast horizons the instant predictor values preserve higher predictability.

The magnitude of improvement is slightly larger in spring and summer seasons, likely because warm-season precipitation is often convective and has more of a diurnal pattern. However, despite the differences in predictor variables among seasons, the best value of $\tau$ is relatively similar across seasons.

Since we determined the optimal values for $\tau$ as a function of season, forecast day, and accumulation window, we apply the best significant $\tau$ value according to Figure 5 and Figure 6 in the following parts of this study.

3.3. Supplemental Lead Times (SLTs)

The SLT approach (described in Section 2.2.2) is tested using windows with up to $\pm 10$ SLTs. At this window width the best out of 21 AnFcst candidates is considered, rather than only the one PaFcst at the target lead time as in the original approach. Since we wish to retain error characteristics from similar lead times, we do not assess the effect of SLTs on discrete windows, nor beyond a lead-time offset of 10 h.

Across stations and forecast days, there is a clear tendency that twCRPSS values improve with increasing SLT window (Figure 7). The steepest improvement occurs within $\pm 3$ SLTs and levels out at $\pm 6$ SLTs on forecast day 3, but day-1 and day-2 average scores keep improving at a decreasing rate until our maximum of $\pm 10$ SLTs. Both, twCRPS and twCRPSS changes are significant in every step of growing SLT windows.

While TTS considerations yield best improvements for longer forecast horizons, the SLT approach benefits shorter forecast horizons most. A reason for this could be that on forecast day 1, the limiting factor in AnEn performance is the availability of good analogs from the provided sample size, whereas on day 3, the limiting factor is the quality of the forecast itself.

3.4. Verification

Not all methods could be performed on the discrete accumulations due to their larger time steps, and the predictor selection was in part already evaluated in Section 3.1; therefore, this verification section focuses on hourly and rolling windows only. Verification is conducted by running the AnEn over the independent 1-year testing period with the best tuning parameters determined in Section 3.1, Section 3.2 and Section 3.3. We show the stepwise improvement by comparing:

• the control AnEn using reference predictors, no TTS, and no SLTs (Control),
• the AnEn with optimized predictors, but no TTS, and no SLTs (Step 1),
• the AnEn with optimized predictors and optimized TTS consideration, but no SLT (Step 2), and
• the AnEn with optimized predictors and optimized TTS consideration, and using SLTs in a window of $\pm 6$ (Step 3).

First, the performance improvement over the raw WRF forecasts is assessed. For comparison we transform the probabilistic AnEn into a deterministic forecast by taking the ensemble median and calculate the mean absolute error (MAE) with the verifying observations. Analogous to Equation (A2) for twCRPSS, the MAE skill score (MAESS) is computed, where positive values represent improvement over the MAE of the raw WRF forecasts.

As seen in Figure 8, all of the AnEn post-processing methods generally have higher rates of improvement for shorter accumulation windows and longer lead times. Even the Control AnEn significantly improves the raw WRF forecasts, and the additional steps further enhance the AnEn performance. For example, the Control AnEn improves WRF MAEs by about 13.3% for hourly forecasts (averaged over forecast days) and about 5.5% for daily forecasts, while the Step 3 AnEn reduces WRF MAEs by an additional 2.9% and 2.0%, respectively. Compared to the Control, our Step 3 AnEn improves hourly-precipitation MAESS by 30.6% on day 1, 26.3% on day 2, and 9.6% on day 3; whereas 12-hourly MAESS are improved by 83.8%, 74.0%, and 41.6%, respectively.

The optimization steps show step-wise improvement, except Step 2 shows a slight drop in MAESS compared to Step 1 for accumulations larger than 3 h. This indicates that the TTS does not always yield the expected improvement as seen during training. The largest improvement is consistently seen in Step 3 from using SLTs.

Recall that the predictor selection was optimized on only forecast day 1 using 3-hourly windows. Yet across accumulation windows and forecast days the same predictors improve the Step 1 MAE, except for daily precipitation on forecast day 1. This study shows results up to daily intervals only for reference, however, users who desire daily-precipitation AnEn forecasts are advised to re-trained the predictor selection to assess whether different variables and weights would make better predictors.

To date, in this paper, we have focused on AnEn improvements for 75p events. Figure 9 and Figure 10 show 90p results to assess if the AnEn forecasts remain skillful for the heavier precipitation events.

The reliability diagrams in Figure 9 show that AnEn probabilities compare well to observed relative frequency across forecast days and accumulation windows; i.e., the AnEns are calibrated and reliable. Relative to the dashed line that represents perfect reliability, most points in the calibration functions have a small deviation towards the left side. This means that the AnEns have a small dry bias for 90p events. This is a common property of AnEns, especially for high-impact events for which only a smaller number of good AnFcsts are available [22,38,48].

Compared to the Control, Step 1 and Step 2 slightly worsen this bias, however, Step 3 moves the calibration function back closer to the line of perfect reliability. This is particularly meaningful because the SLT approach further improves sharpness. Sharper AnEn forecasts were expected as a result of SLTs, because the larger number of considered PaFcsts provides a better chance for the selection of closer AnFcst. These results agree with [54], and they also agree with [38]’s supplemental-locations approach, which uses spatial rather than temporal supplements to inflate the PaFcst sample size.

The receiver operating characteristic (ROC) diagram in Figure 10 shows the discrimination between 90p events and non-events. Larger values of the area under the curve (AUC) score are better, corresponding to a higher true-positive rate (i.e., POD or hit rate) and a lower false-positive rate (i.e., false-alarm rate).

The AnEn AUC scores increase with each optimization step, however, the improvement is larger for shorter accumulations and longer forecast horizons, both of which exhibit worse discrimination to begin with. Although in Figure 8 and Figure 9 Step 2 (TTS) showed smaller improvements or sometimes even worse performance in comparison, TTS contributes considerable improvement with regard to AUC, in particular on forecast day 3 for shorter accumulation windows.

4. Summary and Conclusions

This study demonstrates the benefits of existing and new optimization techniques for AnEn post-processing on sub-daily precipitation forecasts over southwest BC. Lower precipitation rates are easier to predict and have less impact, but since they are far more common they are likely to dominate optimization procedures. Therefore, this study tuned the AnEn parameters for moderate and heavier events based on the 75p twCRPS, instead of the full CRPS as in most other studies.

First, we objectively optimized the choice of predictor variables and their weights by evaluating four variants of forward selection (FS). Since common predictor optimization techniques come at significant computational expense, we suggested the efficient FS (EFS)–an adaptation of [44]’s FS. Limiting the weighting options in sequence with the selected predictors significantly reduces computational cost while maintaining similar optimization performance. Predictor tuning is beneficial even if trained on a portion of the dataset (i.e., only one forecast day instead of the full forecast horizon) and even if the initial set of variables on which the (E)FS is conducted is pre-selected by filter methods such as DCorr. However, EFS on a larger number of meteorological variables can result in minor additional improvements and could be considered in other studies if the computational capacity exists.

Next, we explored the impact of the time-window width over which the temporal predictor trends are matched. This investigation revealed that longer time windows are most beneficial for longer forecast horizons and shorter accumulation windows—a relationship that is often neglected in other studies. Although TTS was shown in the verification Section 3.4 to increase the unconditional dry bias, it improves discrimination of high-impact events. Perhaps the method described in Section 3.2 to assess optimal time windows could be generalized across seasons, such as a staggered implementation for forecast days 1, 2, and 3 using $\tau$ equal to 1 or 2, 2 or 3, and 3 or 4, respectively. Caution is advised when using TTS for discrete windows of accumulations that are longer than hourly, because we obtained mixed results dependent on lead time.

Finally, we implemented a new methodology that uses the concept of supplemental lead times (SLTs). It enhances the chance of finding better AnFcsts by allowing the algorithm to choose from forecast lead times within a time window around the target lead time, that should maintain similar error characteristics. This approach is similar to the idea in [54], but it is suitable for shorter accumulations and prevents the selection of temporally dependent AnFcsts. SLTs could be used in addition to [38]’s supplemental locations, or as an alternative if the domain or station sample size is not sufficient (as in this study).

The use of SLTs had the largest impact on AnEn performance, often exceeding the effects of predictor and TTS optimization, especially for verification statistics including the ensemble-median MAE and 90p reliability and sharpness. The time window width for which SLTs showed performance increase is relatively wide in this study, likely because precipitation in BC has no pronounced diurnal cycle in its cool/wet season. Other predictands with more pronounced diurnal cycle would likely require shorter SLT windows and may experience smaller relative improvements. It is conceivable that a longer dataset history would result in similar improvements, dampening the impact of SLTs. However, when relatively short training periods are available (<5 years in this study), the SLT approach somewhat compensates for the small sample size. This opens up opportunities for AnEn applications on shorter but locally optimized and operational data products.

One NWP model forecast produces three-dimensional multivariate deterministic predictions, whereas the AnEn method in this study creates a univariate probabilistic point forecast by post-processing NWP. Compared to NWP ensembles, AnEns are extremely efficient in creating reliable probabilistic point forecasts—that is, if a sufficiently long reforecast dataset is available. Although our algorithm was not optimized for efficiency, a single 3-day forecast at one point location takes on average only 0.5 s to create the Control or Step 1 AnEn on a macOS computer (with 3.2 GHz Intel Core i5 processor), 1 s to create the Step 2 AnEn (with TTS), and <10 s to create the Step 3 AnEn (with TTS and $\pm 6$ SLTs). This computational time applies to the AnFcst search and AnEn composition only (i.e., excluding time for running the NWP TaFcst and interpolation to station locations) and would have to be multiplied by the number of point locations at which forecasts are desired (if not run simultaneously in parallel). In comparison, the three-domain WRF runs used in this study took on average 80 min run time using 48 cores (Intel-compiled WRF code run on an HPC cluster using Open MPI and no hyperthreading on Intel Xeon Processor E5–2683 v4 compute nodes with 2.10 GHz). While at least one NWP run is required to make an AnEn forecast, running an equivalent-sized 30-member WRF ensemble would require 10 runs, which would take approximately 27 core days of run time in serial mode using configurations (e.g., domain setup) as in [58].

In this study, we improved the computational efficiency of AnEn optimization, while also significantly improving AnEn forecast performance by up to 83.8% compared to a reference AnEn. This increase in AnEn performance can be attributed mainly to a new SLT technique. Temporal variability and forecast-error growth cause finer-temporal-resolution and longer-lead-time forecasts to have inherently worse performance, especially over the complex terrain of southwest BC—yet those forecasts benefit the most from the optimized AnEn post-processing. This is an important result in a world where end users desire evermore accurate predictions at finer resolutions and longer outlooks.