Advances in Imputation Strategies Supporting Peak Storm Surge Surrogate Modeling

Abstract

Surrogate models are widely recognized as effective, data-driven predictive tools for storm surge risk assessment. For such applications, surrogate models (referenced also as emulators or metamodels) are typically developed using existing databases of synthetic storm simulations, and once calibrated can provide fast-to-compute approximations of the storm surge for a variety of downstream analyses. The storm surge predictions need to be established for different geographic locations of interest, typically corresponding to the computational nodes of the original numerical model. A number of inland nodes will remain dry for some of the database storm scenarios, requiring an imputation for them to estimate the so-called pseudo-surge in support of the surrogate model development. Past work has examined the adoption of kNN (k-nearest neighbor) spatial interpolation for this imputation. The enhancement of kNN with hydraulic connectivity information, using the grid or mesh of the original numerical model, was also previously considered. In this enhancement, neighboring nodes are considered connected only if they are connected within the grid. This work revisits the imputation of peak storm surge within a surrogate modeling context and examines three distinct advancements. First, a response-based correlation concept is considered for the hydraulic connectivity, replacing the previous notion of connectivity using the numerical model grid. Second, a Gaussian Process interpolation (GPI) is examined as alternative spatial imputation strategy, integrating a recently established adaptive covariance tapering scheme for accommodating an efficient implementation for large datasets (large number of nodes). Third, a data completion approach is examined for imputation, treating dry instances as missing data and establishing imputation using probabilistic principal component analysis (PPCA). The combination of spatial imputation with PPCA is also examined. In this instance, spatial imputation is first deployed, followed by PPCA for the nodes that were misclassified in the first stage. Misclassification corresponds to the instances for which imputation provides surge estimates higher than ground elevation, creating the illusion that the node is inundated even though the original predictions correspond to the node being dry. In the illustrative case study, different imputation variants established based on the aforementioned advancements are compared, with comparison metrics corresponding to the predictive accuracy of the surrogate models developed using the imputed databases. Results show that incorporating hydraulic connectivity based on response similarity into kNN enhances the predictive performance, that GPI provides a competitive (to kNN) spatial interpolation approach, and that the combination of data completion and spatial interpolation emerges as the recommended approach.

1. Introduction

The prediction of storm surge hazard and its consequences has become a priority when examining the resilience of coastal communities in planning (pre-disaster), emergency management, and post-disaster contexts [1,2,3]. To address this priority, researchers have made substantial efforts to develop high-fidelity numerical models to establish accurate predictions for the anticipated surge levels driven by tropical cyclones and other storm events [4,5]. These models entail, however, a large computational burden, due to the need to resolve complex hydrodynamic processes in the nearshore region [5,6]. To address this challenge, and improve computational efficiency for regional coastal hazard assessment [7,8] or real-time surge forecasting [9,10], researchers have examined the use of surrogate modeling techniques [11,12,13,14,15,16,17,18,19,20]. Surrogate models, also referenced as metamodels or emulators, correspond to data-driven predictive tools developed using a database of simulation results from a high-fidelity numerical model, and offer a computationally efficient approximation of the input/output relationship of that model. In the context of storm surge emulation, regional flood study databases, consisting of surge predictions for synthetic storm scenarios, are typically used for the surrogate model development. Once properly calibrated, these surrogate models can provide highly accurate and computationally efficient predictions with substantially reduced computational cost, and can be employed to either complement the original high-fidelity model for accelerating the hazard estimation [9] or to fully replace that model for any type of future surge predictions [8], accommodating even the development of real-time decision support tools [21].

The surrogate model surge predictions need to be established for different geographic locations of interest, corresponding typically to the computational nodes of the underlying numerical simulation model or to some subset of them. These nodes cover coastal areas consisting of offshore, onshore, and inland locations, and their number may vary from a few thousand up to more than one million. Frequently, a number of inland nodes will remain dry for some of the database storm scenarios, and therefore, a storm surge estimate will not be provided for them by the numerical model. If the intent of a metamodeling implementation is to provide a binary classification of the node condition (i.e., wet or dry), then the information within the database is sufficient for the emulator development [22]. If, on the other hand, the intent is to provide storm surge predictions, an imputation stage is first needed [23] to fill in the missing data with the so-called pseudo-surge [24]. Spatial interpolation, specifically using weighted k-nearest neighbor (kNN) interpolation, has been shown to be effective, especially for the imputing peak storm surge responses [23]. This approach uses the surge values of neighboring inundated nodes to infer the missing data, with the distances to the neighbors (representing spatial correlation) used as weights in the interpolation scheme. The enhancement of kNN by using the connectivity of the underlying numerical grids as a proxy for the hydraulic connectivity was further examined in [22], reflecting the observation that in domains with complex geomorphologies (e.g., ridges and sharp elevation changes) or flood protection measures (e.g., levees, floodwalls, or seawalls), spatial proximity does not necessarily mean correlated surge responses.

This work revisits the imputation of peak storm surge databases when couched within a specific objective—the development of surrogate models for emulation of the peak-surge. This objective dictates both the specific imputation strategies examined, advancing techniques previously considered for this application, as well as the accuracy metrics utilized to evaluate the performance of the different imputation strategies. Also, the focus is specifically on imputation for the peak-surge and not for the surge time-series. The imputation task for these two types of responses has very different characteristics. For example, time-series databases offer richer information due to the inclusion of an additional dimension (the temporal variation), which permits the consideration of an expanded range of imputation strategies that are appropriate for spatiotemporal problems [25,26,27]. In contrast, imputation of peak storm surge values may pose greater challenges, though it has significant relevance due to the wide use of peak storm surge emulation techniques for coastal hazard estimation [8,14,16].

Specifically, three distinct advancements are considered building upon previous efforts in peak storm surge database imputation. For the first advancement, the use of response-based correlation is examined for infusing hydraulic connectivity information within the spatial interpolation, replacing the previously used proxy based on the numerical model grid connections [22]. This replacement constitutes the first novel contribution of the study. The response-based correlation between nodes is defined using the binary response information for the condition of each node (i.e., inundated/wet or not). Only nodes with correlation higher than a specific threshold are considered as candidate neighbors in the kNN implementation. For the second advancement, a Gaussian Process interpolation (GPI) [28] is examined as an alternative (to the previously used kNN [22]) spatial interpolation-based imputation strategy. GPIs have shown great promise in filling in missing values for spatially correlated data [29], but their application to surge imputation has not been explored to date as they have been hindered by computational challenges due to the large size of the databases, originating from the large number of nodes (e.g., millions of nodes per study). Here, a recently established adaptive covariance tapering scheme [30] is adopted to accommodate an efficient GPI implementation for large datasets, and its advantages are examined compared to kNN-based spatial interpolation. This consideration of GPI for the peak storm surge database imputation represents another novel contribution of the study. For the third advancement, imputation using data completion is examined as an alternative to the aforementioned spatial interpolation techniques. Data completion techniques have been shown to be viable candidates for imputing missing data [31,32], especially for large datasets containing sufficient information for the respective technique to learn the underlying patterns. For storm surge applications, they have been recently shown to be effective for imputing surge time-series predictions [26], but their reliability for peak-surge imputations remains an open question. Here this research gap is addressed, establishing the third novel contribution of the study. Specifically, data completion through a latent space projection using probabilistic principal component analysis (PPCA) [33] is examined for imputation, an approach that has demonstrated high scalability and robust performance [34,35] in similar problems and has connections to the principal component analysis-based dimensionality reduction widely used in storm surge emulation [12]. The combination of spatial imputation with PPCA is also considered, with the objective of enriching the information first before performing PPCA. Establishing an appropriate combination approach constitutes the last novel contribution of this study.

The first two advancements, representing spatial interpolation-based imputation strategies are discussed in Section 5, while the remaining advancement, using data completion (or combination), is presented in Section 6. Before elaborating on these advancements, the problem formulation, database used in the case study and peak storm surge surrogate model developments are reviewed in Section 2, Section 3 and Section 4, respectively.

2. Problem Formulation

The development of the surrogate model relies on the availability of a synthetic storm database, typically developed during regional coastal flood hazard studies [36]. These databases consist of n synthetic storms and provide storm surge predictions for a total of n_z nodes within the respective computational domains. The objective of the surrogate model developed utilizing a synthetic storm database is to predict the surge at the available nodes (output) for new storms (input). To support this objective, each synthetic storm is parameterized through the n_x-dimensional vector $x\in {ℝ}^{n_x}$ that will serve as the emulator input, with $x_i$ denoting the ith input component. Additional details on the selection of x can be found in [23] and in Section 3 that discusses the case study database. Superscript notation will be utilized to distinguish the storms in the database, with $x^h$ representing the input vector for the hth storm.

For the ith node within the domain, let $s_i^{lat}$ and $s_i^{lon}$ correspond to the latitude and longitude, $e_i$ to the node elevation, $z_i$ to the peak-surge, and $y_i$ to the inundation state indicator, with $y_i=1$ meaning that the node is wet (inundated) and $y_i=0$ meaning that the node is dry. When the relationship to the input vector needs to be explicit, notation $z_i(x)$ [or $y_i(x)$] will also be used for the surge response [or inundation state] for the ith node. Define $s_i=[\begin{array}{cc}{s}_i^{lat}& s_i^{lon}]\end{array}\in {ℝ}^2$ as the geo-location coordinate vector for the ith node and $S\in {ℝ}^{2\times n_z}$ as the matrix assembling all nodal coordinates. Let $z\in {ℝ}^{n_z}$ denote the $n_z$-dimensional surge vector, with its components corresponding to the surge values for all individual nodes of interest, and let $y\in {ℝ}^{n_z}$ denote the corresponding inundation state vector. All notations including the subscript, superscript, and input dependencies extend to z and y, and to their individual components. For example, $z_i^h=z_i(x^h)$ corresponds to the peak-surge for the ith node for the hth storm [described through the input vector $x^h$] and $z(x^h)$ corresponds to the vector of peak-surge for all nodes for the same storm.

The original database includes missing data, i.e., no information available for the storm surge, when inland nodes remain dry. These correspond to dry instances $y_i^h=0$ and have no corresponding value $z_i^h$. Herein we will reference as always-wet the nodes that have been inundated across the entire database and as once-dry the nodes with at least one missing value, for which $y_i^h=0$ at least once across the storm suite database. We will assume that $n_r$ such nodes (once-dry) exist within the database.

Finally, after the synthetic storm parameterization, the database provides the storm input matrix $X={[x^1 x^2 \dots x^n]}^T\in {ℝ}^{n\times n_x}$ and the output matrix for the surge responses $Z={[z^1 z^2 \dots z^n]}^T\in {ℝ}^{n\times n_z}$. Based on the latter matrix, the binary inundation state condition matrix $Y={[y^1 y^2 \dots y^n]}^T\in {ℝ}^{n\times n_z}$ is derived, with 0 representing the missing surge values in Z (node dry). The hth rows of all these matrices correspond to the characteristics (parametric input or predictions) for the hth synthetic storm and will be distinguished (when needed) by superscript h, with $Z^h$ corresponding, for example, to the row vector with the surge responses for the hth storm. Note that the missing data in the Z matrix ultimately correspond to the elements in $Y$ equal to 0.

3. Database Description

The high-fidelity storm surge database utilized in this study is part of the U.S. Army Corps of Engineers (USACE) Coastal Hazards System (CHS; https://chs.erdc.dren.mil accessed 27 August 2025) [37]. It specifically corresponds to the CHS Louisiana (CHS-LA) coastal flood hazard study [36], developed for quantifying storm hazards and coastal compound flooding in Louisiana. CHS-LA includes areas in the vicinity of the Greater New Orleans Hurricane Storm Damage Risk Reduction System (HSDRRS), which encompasses the district of focus here. A detailed description of the database has been presented in [22]. Here, a basic overview is presented.

The CHS-LA tropical cyclone (TC) suite consists of $n$ = 645 synthetic storms separated into 85 master tracks (MTs), shown in Figure 1. All TCs are characterized by unique combinations of the following parameters: landfall location, defined by the latitude $x_{lat}$ and longitude of the storm’s eye $x_{lon}$, heading direction of the storm track during the final approach to landfall $\beta$, central pressure deficit $\Delta P$, forward translation speed $v_t$, and radius of maximum wind speed $R_{mw}$. Each MT is initially constructed as a unique combination of the storm’s heading direction $\beta$ and its landfall location $[x_{lat} x_{lon}]$. The remaining parameters defining the synthetic storms dictate the strength $\Delta P$, size $R_{mw}$, and translational speed $v_t$ characteristics of each synthetic storm, further distinguishing storms that might correspond to the same MT.

Table 1. Storm characteristics per master track (MT) group for CHS-LA.

ID	Storm Heading β (°)	Number of Different MTs (Landfall Locations)	Number of Storms per MT	Storm Parameter Range
				$\mathbf{\Delta }\mathit{P}$ (mbar)	${\mathit{R}}_{\mathit{m}\mathit{w}}$ (km)	${\mathit{v}}_{\mathit{t}}$ (m/s)
1	−80	6	48	[8 148]	[9.3 115.5]	[2.4 10.60]
2	−60	9	70	[18 138]	[11.8 127.5]	[2.4 12.50]
3	−40	13	104	[8 148]	[8.5 133.1]	[2.2 13.90]
4	−20	15	105	[18 138]	[9.1 116.5]	[2.4 11.80]
5	0	15	120	[8 148]	[8.0 130.0]	[2.4 13.05]
6	20	14	98	[18 138]	[8.6 138.2]	[2.3 12.65]
7	40	9	72	[8 148]	[9.6 141.3]	[2.3 12.85]
8	60	4	28	[18 138]	[9.4 119.4]	[2.5 13.40]

summarizes the range of the TC parameters that constitute the database.

The simulation of the 645 synthetic TCs was performed using high-resolution, high-fidelity atmospheric and hydrodynamic numerical models. The forcing of these models for each storm corresponded to the wind and pressure fields obtained using a nested grid Planetary Boundary Layer (PBL) model with input defined through the parametric description of the storm. The hydrodynamic simulations were performed by coupling the ADCIRC [ADvanced CIRCulation model] [38] and the STWAVE [Steady-State Spectral Wave model] [39]. The ADCIRC mesh consisted of close to 1.6 million nodes and 3.1 million triangular elements. The geographic domain includes various flood protection systems around the New Orleans area. A subset of the entire domain will be considered for the metamodel development, focusing on areas around New Orleans, constrained by latitude [28.5°, 40°] N and longitude [86°, 93.5°] W. This corresponds to a total of $n_z=$1,179,179 nodes, with $n_r=$ 488,216 being dry in at least one storm. Figure 2 presents the histogram of the percentage of storms for which each node is inundated. Note that percentage equal to 100% corresponds to the set of always-wet nodes. The spatial distribution of the $n_r$ nodes with the percentage of storms for which these nodes are inundated will be illustrated later in Section 7.1.

With respect to the input parameterization, as shown in [22] the storm strength, size, and translational speed, characteristics for the CHS-LA suite have been held relatively constant starting at approximately 250 km prior to landfall. This incentivizes the use of the characteristics at a reference landfall location to define $x$. Following the recommendations in [13], a piece-wise linear coastal boundary is utilized for defining this reference landfall location to avoid ambiguous definition of landfall due to the existence of bays. This boundary is also shown in Figure 1 (white solid line). The reference landfall is defined at the intersection of this boundary and the synthetic storm track. A single parameter is adopted for describing this landfall, corresponding to the distance (in km) of the landfall location along this boundary from a reference point (any reference point can be utilized here). This leads to a five-dimensional ($n_x$ = 5) input vector that includes the distance for the reference landfall along the linearized boundary, along with $\beta$, $\Delta P$, $R_{mw}$, and $v_t$ at final approach to landfall.

4. Surrogate Modeling Overview

This section briefly reviews different aspects of the surrogate model implementation: (i) the database imputation; (ii) the development and calibration of two surrogate models, one for regression and one for classification, and their combination to establish the final storm surge predictions; and (iii) the validation of the established models. Additional details can be found in [22]. Readers familiar with the surrogate model implementation may proceed directly to Section 5 and Section 6.

4.1. Database Imputation

The development of the regression surrogate model (discussed in Section 4.2) requires imputation of the original database to provide the pseudo-surge [24] for any missing data for Z. For the ith node and hth storm, this pseudo-surge is denoted as ${\underset{˜}{z}}_i^h$ and can be obtained through any of the approaches discussed in detail in Section 5 and Section 6. As discussed in detail in [23], imputation is not guaranteed to lead to pseudo-surge values ${\underset{˜}{z}}_i^h$ that are smaller than the node elevation $e_i$, with nodes for which ${\underset{˜}{z}}_i^h>e_i$ falsely classified as wet based on the imputed surge values. The solutions for accommodating such erroneous information in the surrogate model development are as follows [23]: (a) adjust the estimated ${\underset{˜}{z}}_i^h$ to be smaller (for example 10 cm smaller) than $e_i$, guaranteeing that ${\underset{˜}{z}}_i^h<e_i$ and the estimated pseudo-surge provides a correct classification of the node as dry; or (b) retain the (erroneously classified) pseudo-surge values in the development of the regression surrogate model but supplement that regression model with an additional surrogate model for the classification of the inundation condition of each node, established using the original condition matrix Y (which does not include any erroneous information). The term problematic nodes and notation A_pn are introduced herein to describe the subset of nodes that are at least once misclassified during the imputation process. Following [22], the terminology pseudo-surge database and corrected pseudo-surge database will be used to describe, respectively, the imputed database with no modification [case (b)] or with adjustment that guarantee all imputed values lead to correct classification of nodes [case (a)].

As discussed in detail in [22,23], the corrected pseudo-surge database includes discontinuities (jumps) in the imputed observation matrix Z. These discontinuities are represented by the surge gap ${\eta }_i$ for the once-dry nodes, defined using the information in the original database as follows [22]:

$${\eta }_i={\min }_h(z_i^h)-e_i$$

(1)

where ${\min }_h(.)$ denotes the minimum of the quantity inside the parentheses across all the storms in the database. The larger the value of this gap, the greater the challenges for the regression surrogate model in achieving accurate predictions when the corrected pseudo-surge database is utilized. As discussed in detail in [23], this is the main reason for promoting the use of the pseudo-surge database in the surge surrogate model development. It is expected to provide a higher accuracy predictive model, as long as the erroneous information in this database (originating from the imputed data) can be corrected when supplementing predictions with the ones from the secondary classification surrogate model.

To assess the quality of information associated with the imputed database, the following measures are introduced. The maximum surge misinformation $m{s}_i$, corresponding to the largest difference between the imputed pseudo-surge value and the node elevation:

$$m{s}_i={\max }_h\{({\underset{˜}{z}}_i^h-e_i)\cdot |y_i^h-1\left|\right\}$$

(2)

and the percentage of misinformation $m{p}_i$, corresponding to the relative ratio of storms (with respect to the total database) that a node is misclassified at the imputation stage:

$$m{p}_i={\displaystyle \frac{1}{n}}{\displaystyle \sum _{h=1}^n\{\mathbb{I}[{\underset{˜}{z}}_i^h>e]\cdot |y_i^h-1\left|\right\}}$$

(3)

where $\mathbb{I}[.]$ denotes the indicator function, corresponding to one if the expression inside the brackets is true and to zero otherwise. Note that the term $|y_i^h-1|$ restricts the operations in the above equations to only the originally dry instances, i.e., when $y_i^h=0$. For each problematic node, the maximum surge misinformation and the percentage of misinformation represent, respectively, the magnitude or amount of erroneous information. A modification of the problematic node definition was proposed in [22] using $m{s}_i$ and $m{p}_i$. Any problematic nodes that correspond to small values for both $m{s}_i$ and $m{p}_i$ can be removed from the set $A_{pn}$; the information quality for them is deemed as sufficiently high (erroneous information is negligible). This modified problematic node set is denoted as ${\overline{A}}_{pn}$ herein. In the case study presented later, the thresholds used for $m{s}_i$ and $m{p}_i$ are 5 cm and 2.0%, respectively, following the recommendation in [22].

4.2. Surrogate Model Development

4.2.1. Overview of Data-Driven Predictive Model Formulation

The storm surge predictive model is composed of two different surrogate models [22]: (a) the primary regression surrogate model, denoted as S_s herein, for approximating the storm surge $z(x)$; and (b) the secondary classification model, denoted as S_c herein, for approximating the inundation state $y(x)$. The regression surrogate S_s is established using a database with input and output matrices corresponding to X and imputed Z, respectively. Either the pseudo-surge database or the corrected pseudo-surge database can be used for Z. The classification surrogate S_c is established using a database with input and output matrices corresponding to X and Y, respectively. Note that the classification surrogate can be established only for the once-dry nodes [total of $n_r$ nodes] since for the always-wet nodes there is no useful information in the database to support a classification metamodel development (all training points correspond to the same value of 1). The metamodel predictions will be denoted as ${\tilde{z}}_i(x)$ and ${\tilde{y}}_i^c(x)$ for the ith node and as $\tilde{z}(x)$ and ${\tilde{y}}^c(x)$ for the entire output vector, for the S_s and S_c surrogate models, respectively. Note that the S_s metamodel also establishes predictions for the inundation state, denoted by ${\tilde{y}}_i^s(x)$ for the ith node and given by the following:

$${\tilde{y}}_i^s(x)=\mathbb{I}[{\tilde{z}}_i(x)>e_i]$$

(4)

where, as defined earlier, $\mathbb{I}[.]$ denotes the indicator function. The predictions for the entire output vector, with elements ${\tilde{y}}_i^s(x)$, will be denoted as ${\tilde{y}}^s(x)$. Note that superscripts c and s were introduced to distinguish the predictions for the node inundation state according to the S_c and S_s surrogate models, respectively. The final approximation for the inundation state, denoted as ${\tilde{y}}_i(x)$ for the ith node and as $\tilde{y}(x)$ for the entire output vector, can be obtained, as will be discussed in Section 4.2.3, by the appropriate combination of the predictions of each of these metamodels.

4.2.2. Classification and Regression Surrogate Model Details

Any type of surrogate model [15,16] can be utilized for establishing the S_s and S_c metamodels. In the illustrative case study, a Gaussian Process (GP) metamodel [40,41] is considered as the surrogate model for both S_c and S_s, coupled with a dimensionality reduction step, using principal component analysis (PCA), to deal with the potentially large dimension of the output (i.e., large number of nodes). This implementation has been shown to be highly efficient for storm surge surrogate model predictions [12,23]. Details for the surrogate models are presented in Appendix A and Appendix B for S_s and S_c, respectively, with Appendix C providing an overview of the GP formulation. This surrogate modeling implementation establishes ultimately probabilistic predictions for the node inundation state. Instead of providing a binary classification, i.e., predicting node as wet or dry, these predictions provide the probability of the node being inundated. For the ith node these predictions will be denoted herein as $p_i^s(x)$ and $p_i^c(x)$ for the S_s and S_c surrogate models, respectively. The binary classification can be established as $y_i^s(x)=\mathbb{I}[p_i^s(x)>0.5]$ and $y_i^c(x)=\mathbb{I}[p_i^c(x)>0.5]$, respectively, though the probabilities can be explicitly utilized in the combination of the surrogate model predictions, as detailed next.

4.2.3. Combination of Individual Surrogate Model Predictions

The final storm surge surrogate model predictions are established by combining the S_s and S_c metamodels, with both of them leveraged to establish the predictions for the inundation state ${\tilde{y}}_i(x)$, and the first one [S_s] used to provide the final surge predictions ${\tilde{z}}_i(x)$ for the nodes that are predicted to be inundated, i.e., correspond to ${\tilde{y}}_i(x)=1$. A detailed discussion on alternative options for the metamodel combination is presented in [22]. Here, a quick review of the recommended implementation is provided. To accommodate this combination, the following classes of nodes are defined:

• The always-wet nodes (inundated across the entire database), with no missing information. These will be denoted as $C^1$ class. The node condition classification for them is based entirely on S_s.
• The modified problematic nodes ${\overline{A}}_{pn}$ for which imputation was needed, and that imputation led to sufficiently large values of $m{s}_i$ or $m{p}_i$. These will be denoted as $C^2$ class.
• The remaining nodes, for which imputation was needed, but that imputation did not lead to significant misclassification. These will be denoted as $C^3$ class.

Additionally, the combined surrogate model probabilistic predictions are defined as follows:

$$p_i^{cb}(x)=w_i^{cb}{p}_i^s(x)+(1-w_i^{cb})p_i^c(x)$$

(5)

where $0\le w_i^{cb}\le 1$ and $(1-w_i^{cb})$ correspond to the weights assigned to the S_s (regression) and S_c (classification) metamodels. Note that if the established metamodels for S_c and S_s do not provide probabilistic predictions, the binary classification can be utilized in the combination. The selection of weights $w_i^{cb}$ reflects the degree of confidence for each metamodel class with a balanced implementation recommended in [22]. The implementation weighs equally ($w_i^{cb}=1/2$) the S_s and S_c predictions for all instances that these predictions could be deemed as reliable. This reliability should be evaluated under the following consideration. S_s predictions for class $C^2$ are biased towards false positives (dry nodes characterized as wet) due to the underlying erroneous information in the pseudo-surge database (see also Section 4.1 discussion). Based on this tendency, if S_s predictions classify nodes as dry, they can be assessed as reliable, as they are opposite to the underlying metamodel bias. In such cases, both models can be combined to provide $p_i^{cb}(x)$. On the other hand, if S_s predictions classify nodes as wet, i.e., ${\tilde{z}}_i(x)\ge e_i$, then due to the underlying false positive bias they should be ignored as not credible. The recommendation for the surrogate model combination is finally as follows:

$${\tilde{y}}_i(x)=\{\begin{array}{c}{\mathrm{C}}^1:\hspace{1em}{y}_i^s(x)=\mathbb{I}[{\tilde{z}}_i(x)>e_i]\hfill \\ {\mathrm{C}}^2:\{\begin{array}{c}\mathbb{I}[p_i^{cb}(x)>0.5|w_i^{cb}]=0\hspace{1em}\mathrm{if}\hspace{1em}{\tilde{z}}_i(x)<e_i\hfill \\ y_i^c(x)=\mathbb{I}[p_i^c(x)>0.5]\hspace{1em}\hspace{1em}\hspace{1em}\mathrm{else}\hfill \end{array}\hfill \\ {\mathrm{C}}^3:\hspace{1em}\mathbb{I}[p_i^{cb}(x)>0.5|w_i^{cb}]\hfill \end{array}$$

(6)

It should be noted that for the S_s surrogate, the relationship $y_i^s(x)=\mathbb{I}[{\tilde{z}}_i(x)>e_i]$$=\mathbb{I}[p_i^s(x)>0.5]$ also holds.

Finally, the surge predictions are established as follows. For the instances corresponding to ${\tilde{y}}_i(x)=1$, the surge estimates are provided directly by the S_s metamodel, ${\tilde{z}}_i(x)$, as long as ${\tilde{z}}_i(x)>e_i$. If ${\tilde{z}}_i(x)<e_i$ and the node would have been classified eventually as dry, even if the combined surrogate model predictions yield ${\tilde{y}}_i(x)=1$, then instead of using directly ${\tilde{z}}_i(x)$ the predictions are set equal to some margin (2.0 cm used in this study) over the node elevation $e_i$. This is necessary so that assigned surge predictions correspond to nodes classified as inundated for all ${\tilde{y}}_i(x)=1$ instances.

4.3. Surrogate Model Validation

For validation, a k-fold cross-validation (CV) is adopted. The process for establishing it is as follows: (s.1) the original database is randomly partitioned to k different and equally sized groups; (s.2) sequentially, each group is removed from the database, and the surrogate model is calibrated using as observations the remaining storms; (s.3) predictions for the surge of the removed storms are established using this surrogate model; (s.4) comparing these predictions to the actual storm output (of the removed storms) quantifies the prediction error for them; (s.5) accuracy statistics are estimated by combining error information across all folds. Note that when imputation relying on data completion is utilized for the metamodel development, then imputation is repeated for each k-fold in step (s.2). The k-fold CV implementation ultimately provides for the ith node estimates for the surge ${\tilde{z}}_i(x^h)$ and inundation condition ${\tilde{y}}_i(x^h)$ established for the hth storm by using astraining database the specific group that does not include the storm.

As validation metrics, the misclassification and surge score are utilized. For a specific node and storm, the total misclassification is given by the following:

$$M{C}_i^h=\left|{\tilde{y}}_i(x^h)-y_i^h\right|.$$

(7)

This total misclassification can be decomposed to the false positive and false negative components, with the first corresponding to the instances that the node is predicted wet when dry, and the second to the instances the node is predicted dry when wet. These misclassification types are described, respectively, by the following equations:

$$\begin{array}{l}{}^{+}MC_i^h=\max (0,{\tilde{y}}_i(x^h)-y_i^h)\\ {}^{-}MC_i^h=\max (0,y_i^h-{\tilde{y}}_i(x^h))\end{array}$$

(8)

with max(a,b) corresponding to the maximum of the two arguments a or b. Average misclassification statistics per node or across all nodes can be then estimated to describe the accuracy. These are separately estimated for the false positive/false negative/total misclassification components. For the latter they are denoted as $M{C}_i$ for the statistics per node and as $\overline{MC}$ for the statistics across all nodes (across entire database) and are given, respectively, by the following:

$$M{C}_i={\displaystyle \frac{1}{n}}{\displaystyle \sum _{h=1}^{n}M{C}_i^h}; \overline{MC}={\displaystyle \frac{1}{n{n}_z}}{\displaystyle \sum _{h=1}^n{\displaystyle \sum _{i=1}^{n_z}M{C}_i^h}}.$$

(9)

The equations for the respective statistics for the false positive or the false negative misclassification are identical with the only difference that instead of $n_z$, the number of nodes that were dry or wet is utilized, respectively, for each storm for the false positive or negative misclassification statistics. Additionally, statistics can be provided for specific groups instead of the entire database; this is accomplished by examining in the averaging in Equation (9) only the nodes belonging to that group (i.e., summation is performed only with respect to the indices i included that group).

The surge score for a specific node and storm is given by the following [10,24]:

$$S{C}_i^h=\left\{\begin{array}{c}|{\tilde{z}}_i(x^h)-z_i^h|\\ {\tilde{z}}_i(x^h)-e_i\\ z_i(x^h)-e_i\\ 0\end{array}\right.\begin{array}{c} \mathrm{if} {\tilde{z}}_i(x^h)>e_i \& z_i^h>e_i \\ \mathrm{if} {\tilde{z}}_i(x^h)>e_i \& z_i^h\le e_i\\ \mathrm{if} {\tilde{z}}_i(x^h)\le e_i \& z_i^h>e_i\\ \mathrm{if} {\tilde{z}}_i(x^h)\le e_i \& z_i^h\le e_i\end{array}=\left|\max (e_i,{\tilde{z}}_i(x^h))-\max (e_i,z_i^h)\right|.$$

(10)

This surge score quantifies the absolute discrepancy between the predicted and actual surge, further incorporating the node inundation state: if the node is wet and is also predicted wet, then the absolute value of the predicted surge discrepancy is used as a penalty function; if the node is wet, but it is predicted dry, or vice versa, then the difference between surge (or predicted surge) and node elevation is used as penalty; if the node is actually dry and it is predicted dry, then the penalty is zero. Note that the expression $|\max (e_i,{\tilde{z}}_i(x^h))-\max (e_i,z_i^h)|$ represents the compact mathematical representation of the four-branch definition of the surge score. The averaged statistics per node, or across the entire database, can be then obtained, respectively, as the following:

$$S{C}_i={\displaystyle \frac{1}{n}}{\displaystyle \sum _{h=1}^{n}S{C}_i^h};\hspace{1em}\overline{SC}={\displaystyle \frac{1}{n{n}_z}}{\displaystyle \sum _{h=1}^n{\displaystyle \sum _{i=1}^{n_z}S{C}_i^h}}.$$

(11)

Statistics for the surge score can be examined for specific groups instead of the entire node set, similar to the misclassification case.

5. Imputation Based on Spatial Interpolation

Spatial interpolation is a popular and practical tool for estimating spatially varying variables for geographical locations where they have not been observed, based on observations from neighboring locations [28]. For the application examined here, spatial interpolation techniques can be calibrated utilizing the always-wet nodes and can be subsequently utilized to estimate the pseudo-surge values (i.e., impute the database) based on the storm surge values of other inundated nodes in close proximity. Imputation is separately performed for each storm. As discussed in the introduction, two separate spatial interpolation techniques are examined: weighted kNN and GPI. Weighted kNN has been previously considered for imputation of peak storm surge databases [22,23], and here an extension is proposed integrating a response-based correlation measure to better describe the hydraulic connectivity between nodes. GPI is an attractive spatial interpolation approach [28,42] but, as discussed in Section 1, to date it has not been considered for the application examined here due to computational challenges associated with the large dimension of the database.

5.1. Imputation Using kNN with Enhanced Hydraulic Connectivity

To formalize the kNN implementation, define $d_{ij}=d(s_i,s_j)$ as the geo-distance between nodes i and j and $A_k^h[i]$ as the set of k closest nodes to the ith node for the hth storm. Importantly, the only nodes included in set $A_k^h[i]$ correspond to nodes with known surge values; these correspond to originally inundated nodes for the hth storm (nodes for which $y_i^h=1$ in the original database) and nodes with already imputed values based on the iterative implementation proposed in [23] (and also reviewed next). The pseudo-surge estimate ${\underset{˜}{z}}_i^h$ for the ith node and the hth storm based on the weighted kNN interpolation is given by the following:

$${\underset{˜}{z}}_i^h={\displaystyle \frac{{\displaystyle \sum _{j\in A_k^h[i]}w(d_{ij})z_j^h}}{{\displaystyle \sum _{j\in A_k^h[i]}w(d_{ij})}}}$$

(12)

where $w(d_{ij})$ is a distance-dependent weight taken as a power exponential with a cut-off distance $d_t$:

$$w(d_{ij})=\left\{\begin{array}{c}{e}^{-{\left({\displaystyle \frac{d_{ij}}{{\psi }_l}}\right)}^{{\psi }_e}} \mathrm{if} d_{ij}<d_t\hfill \\ 0\hspace{1em}\hspace{1em}\hspace{1em} \mathrm{if} d_{ij}\ge d_t\hfill \end{array}\right..$$

(13)

The set of $[k d_t {\psi }_l {\psi }_e]$ corresponds to the kNN hyper-parameters. These can be calibrated to optimize interpolation accuracy across the always-wet nodes [23]. The calibration procedure is reviewed in Appendix D.

The kNN-based imputation is progressively performed [23] within an iterative implementation. At each iteration, only dry nodes that have at least k wet (both imputed and genuine) neighbors in a larger set of k_c nodes are imputed. For nodes for which fewer than k wet neighbors are available, imputation is not performed at the current iteration (these will be imputed in a future iteration). The value of k_c is chosen equal to twice the value of k in this study, adopting the same recommendation as in [23]. Additionally, the use of hydraulic connectivity information within the kNN imputation was introduced in [22] and shown to be advantageous. Instead of the closest nodes based purely on $d_{ij}$, the closest connected neighbors are used, defined by the number of element edges between nodes i and j in the graph representing the underlying numerical grid [for example, ADCIRC in the database discussed in Section 3]. Nodes corresponding to fewer edge connections are given higher priority in estimating the k nearest neighbors independent of the actual distance. This approach ultimately prioritizes relatively ADCIRC connectivity over spatial proximity in selecting the closest neighbors.

The edge connections within the underlying numerical grid offer, though, only a proxy for the hydraulic connectivity between nodes. Here, this concept is advanced to use the inferred connectivity based on the response database Y. Given the fact that the responses are binary, the cosine similarity is adopted as a response-based correlation measure [43], defined for nodes i and j as follows:

$$C_s(i,j)={\displaystyle \frac{{\displaystyle \sum _{h=1}^n{y}_i^h}{y}_j^h}{\sqrt{{\displaystyle \sum _{h=1}^n{(y_i^h)}^2}}\sqrt{{\displaystyle \sum _{h=1}^n{(y_j^h)}^2}}}}.$$

(14)

This cosine similarity represents the normalized degree of inundation state alignment between nodes i and j, with a value equal to 1 representing perfect alignment and a value equal to 0 representing perfect misalignment. Another way to interpret the cosine similarity for the binary variables is the following. Consider the resultant partitioning of the outcome of the binary vectors $\{y_i^h;h=1,\dots ,n\}$ and $\{y_j^h;h=1,\dots ,n\}$ with the following: N₁₁ representing the total number of storms where both $y_i^h$ and $y_j^h$ are equal to 1; N₁₀ representing the total number of storms where $y_i^h=1$ and $y_j^h=0$; N₀₁ representing the total number of storms where $y_i^h=0$ and $y_j^h=1$; and N₀₀ representing the total number of storms where both $y_i^h$ and $y_j^h$ are equal to 0. Then the cosine similarity can be expressed as follows:

$$C_s(i,j)={\displaystyle \frac{N_{11}}{\sqrt{N_{11}+N_{10}}\sqrt{N_{11}+N_{01}}}}.$$

(15)

Alternative measures for quantifying correlation between binary-valued vectors, for example, Jaccard similarity or Hamming similarity, could also be considered instead of the cosine similarity. Finally, a threshold $C_s^t$ is imposed for the cosine similarity with nodes considered connected only if $C_s(i,j)>C_s^t$. The nearest neighbors for node i are finally chosen only from the candidate connected neighbors. This implementation avoids creating connections between nodes with close spatial proximity that do not appearto be correlated based on the response database Y.

5.2. Imputation Using GPI with Adaptive Covariance Tapering

The GPI is established based on the formulation of Appendix C for input and output choices $u=s$ [node spatial coordinates] and $v=z^h$ [spatially varying surge for each storm], respectively, leading to $n_u=2$. It is separately implemented for each storm. To formalize the implementation, let $i_w^h$ and $i_r^h$ represent the indices corresponding to the wet and dry nodes for the ith storm, with the number of elements in $i_w^h$ denoted as $n_w^h$ and the number of elements in $i_r^h$ as $n_r^h$. The training data for the hth storm are $n_w^h$, with input matrix $U$ and output vector $V$ (based on Appendix C terminology) corresponding to the columns of S and $Z^h$ for indices $i_w^h$, with $n_{GP}=n_w^h$. The metamodel predictions given by Equation (A6) for each of the inputs corresponding to the remaining columns of S, for indices $i_d^h$, provide the desired pseudo-surge estimates. Ultimately, this corresponds to predictions for $n_d^h$ new inputs. Similarly to the kNN implementation, a progressive imputation can be achieved for each storm instead of estimating all pseudo-surge values in one iteration. A common correlation kernel used for spatial interpolation is the Matérn function [28,44], expressed for nodes i and j as a function of their distance $d_{ij}=d(s_i,s_j)$ and given by the following:

$$R(s_i,s_j|\gamma )={\displaystyle \frac{1}{2^{v-1}\mathrm{\Gamma }\left(v\right)}}{\left({\displaystyle \frac{d_{ij}}{d_r}}\right)}^v{K}_v\left({\displaystyle \frac{d_{ij}}{d_r}}\right)$$

(16)

where $K_v$ is the modified Bessel function (of the second kind) of order $v$, and $d_r$ controls the rate of decay with distance while $v$ controls the GP smoothness. The hyper-parameter vector of the correlation kernel is $\gamma =[d_r v]$. Note that the correlation kernel is strictly a function of the distance between nodes, i.e., $R(d_{ij}|\gamma )=R(s_i,s_j|\gamma )$.

The challenge in the GPI implementation originates from the large number of wet nodes for each storm, since formulation (see details in Appendix C) requires the inversion of a correlation matrix R [with elements $R(d_{ij}|\gamma )$] of dimension $n_{GP}=n_w^h$. This task has computational complexity of order $O[n_{GP}^3]$ [41]. A variety of solutions [45,46,47,48] have been established for promoting computationally efficiency for GP implementations dealing with such large datasets. For spatial interpolation, covariance tapering is an especially appealing solution due its intuitive interpretation, as it relies on spatial correlation/connectivity between nodes. Additionally, some of its variants [46] are appropriate for irregular grids, like the ones encountered for storm surge surrogate modeling applications. It should be noted that the foundation of covariance tapering resembles the one for kNN: only a reduced number of (close proximity) neighbors should impact the GPI.

Mathematically, covariance tapering leverages inversion algorithms for sparse matrices [49] to reduce the computational burden of inverting large-dimensional covariance matrices. Sparsity is infused by setting unimportant correlations to zero in the GP correlation matrix R, such that the resultant matrix R consists of a very large number of zeros. A smooth taper function with compact support $0\le T(d_{ij}|{\phi }_i,{\phi }_j)\le 1$ is introduced to achieve this objective, with ${\phi }_i$ representing the hyper-parameter defining the compact support for each node, the so-called taper range (or taper length). When the nodal distance exceeds the compact support (defined through the respective taper ranges), zero correlation is presumed. By replacing $R(d_{ij}|\gamma )$ with $R(d_{ij}|\gamma )\cdot T(d_{ij}|{\phi }_i,{\phi }_j)$ the desired sparsity in R can be achieved. To distinguish the resultant sparse matrix when tapering is used, the notation R_s will be used herein. Let ρ and ${\rho }_i$ represent the degrees of global and local sparsity, respectively. These correspond, respectively, to the proportion of non-zero entries in the overall matrix R_s, or in its ith column. The computational cost of inverting R_s for large degrees of sparsity is reduced to approximately $O(n_{GP}\cdot {(\rho n_{GP})}^3)$ [50]. Note that the exact computational burden depends on the details of the implementation. The taper range can be selected to achieve the desired degree of global sparsity to achieve some targeted reduction in the computational burden.

The simplest implementation of covariance tapering is a constant tapering approach using the same taper range across all nodes ${\phi }_i=\phi$ [48], chosen based on the aforementioned target global sparsity ρ. Unfortunately for irregular grids, like the ones frequently encountered in storm surge applications [38], the constant tapering approach promotes non-uniform local sparsity, with some nodes having too few neighbors and others having too many, something that can adversely affect predictive accuracy. An adaptive covariance tapering formulation [46], explicitly trying to achieve the target local sparsity for each node, circumvents this challenge. An adaptive taper is utilized in this case for which ${\phi }_i\ne {\phi }_j$ for i ≠ j. An example of an adaptive taper, and the one utilized in this study, is as follows [46]:

$$T(d_{ij}|{\phi }_i,{\phi }_j)={\displaystyle \frac{1}{\pi R_{ij}{r}_{ij}}}\left\{\begin{array}{cc}\pi r_{ij}^2& \mathrm{if} d_{ij}<R_{ij}-r_{ij}\\ V_2(R_{ij},{\displaystyle \frac{d_{ij}^2+R_{ij}^2-r_{ij}^2}{2d_{ij}}})+V_2(r_{ij},{\displaystyle \frac{d_{ij}^2+r_{ij}^2-R_{ij}^2}{2d_{ij}}})& \mathrm{if} R_{ij}-r_{ij}\le d_{ij}<R_{ij}+r_{ij}\\ 0& \mathrm{otherwise},\end{array}\right.$$

(17)

where $r_{ij}=\min ({\phi }_i,{\phi }_j)/2$, $R_{ij}=\max ({\phi }_i,{\phi }_j)/2$, and

$$V_2(r,x)=\left\{\begin{array}{cc}{r}^2{\cos }^{-1}\left(x/r\right)-x\sqrt{r^2-x^2}& \left|x\right|<r\\ 0& \mathrm{else}.\end{array}\right.$$

(18)

This selection leads to compact support $d_{ij}\le ({\phi }_i+{\phi }_j)/2$ for the taper, and to a non-stationary, sparse covariance matrix that can have uniform local sparsity per node (i.e., constant ${\rho }_i$) even for highly irregular datasets.

A key aspect in the implementation of adaptive covariance tapering is the selection of the taper ranges per node $\{{\phi }_i;i=1,\dots ,n_z\}$ to achieve the desired local sparsity $\{{\rho }_i;i=1,\dots ,n_z\}$. An iterative optimization was established in [46] to support this selection, but that procedure cannot scale to large datasets (beyond a few thousand nodes). Recently [30], a computationally efficient implementation of adaptive covariance tapering was established that can seamlessly scale to applications even with a few hundred thousand nodes. This is the formulation adopted in this study for performing imputation using GPI with adaptive covariance tapering. According to this formulation, a small subset of nodes, termed inducing points, is chosen for guiding the identification of the taper ranges. The ranges are only explicitly chosen for the inducing points, something that greatly simplifies the corresponding optimization, achieving the desired computational efficiency. The taper ranges for the remaining points are inferred from the taper range values chosen for the inducing points. The corresponding algorithm established in [30] for the identification of the taper ranges, termed M-IAT (iterative adaptive taper selection), is reviewed in Appendix E. M-IAT can only directly control the local sparsity of the inducing points, with the local sparsity for the remaining points influenced indirectly by the distribution of those inducing point set. The selection of the inducing points becomes, therefore, critical so that the decisions made based on the inducing points are compatible with the decisions that would have been made if all the points were used. An adaptive, iterative selection was proposed in [30] for the inducing points, and is the one adopted here. This selection starts with a small number of inducing points, performs the taper identification utilizing them (step 1), and then augments them with additional points (step 2) corresponding to regions for which the discrepancy from the target local sparsity (based on the step 1 tapers) is the largest. The latter points represent the ones (among the remaining points not already chosen as inducing points) with the expected largest information value in guiding the taper range selection for achieving the target local sparsity throughout the entire domain. Steps 1 and 2 are repeated until convergence, defined as establishing a discrepancy from the target local sparsity below a specific threshold, is achieved. The corresponding algorithm established in [30] for selecting the inducing points, termed IIP (iterative inducing point identification), is reviewed in Appendix F.

5.3. Computational Complexity and Scalability of Spatial Interpolation Techniques

The computational complexity of kNN is of order $O(h(n_{z}k)+n_z^2)$ if the implementation relies on the estimation of distances and cosine similarity measures between all pairs of nodes. With some intelligent implementation to restrict this estimation to only a smaller number of $k_{pr}$ candidate neighbors (for example within a rectangular domain centered at each node), this complexity may be reduced to $O(h(n_{z}k)+n_z{k}_{pr})$. As such the computational burden of kNN may be considered to scale practically linearly with the size of the database (either storms or nodes), showing that it can be easily extended to larger datasets. On the other hand, the computational complexity of GPI is of order $O(n_z{(\rho n_z)}^3+h{n}_z^2)$. Depending on the adopted value of ρ, this means that the computational burden of GPI scales linearly with the number of storms and at best quadratically with the number of nodes. As such, extension to very large datasets is expected to face challenges, depending on the available computational resources. A partial remedy to this challenges can be provided by employing GPU resources for the relevant matrix manipulations [51].

Note that the iterative implementation of the spatial interpolation techniques to support the progressive imputation of the database will increase the aforementioned computational costs, but the exact cost (or total number of iterations) is database dependent as it depends on the exact pattern of the missing data.

6. Imputation Based on Data Completion

6.1. Motivation for Imputation Utilizing Data Completion

Instead of trying to establish a predictive model by inferring spatial correlations, imputation approaches based on data completion utilize solely the original observation matrix Z. They infer patterns based on all observed data and then project these patterns into the missing data to accomplish the imputation task. As they do not utilize any spatial information, they avoid the pitfalls that might exist for spatial interpolation approaches, when spatial proximity does not necessarily translate into surge response correlation. For imputation of storm surges, they will be implemented utilizing all data, trying to infer patterns across storms and between nodes, to fill in the missing values.

A range of imputation methods relying on data completion exist; for example, variants of low-rank matrix completion [52,53] or principal component analysis (PCA) [54,55]. In general, such approaches perform well when there is sufficient information in the original data to infer the underlying patterns. For peak-surge imputation, they are expected to face challenges due to the limited information that is available for some nodes. This is evident in the results shown in Figure 2 for the case study database, with a significant portion of the nodes (over 15%) being inundated in less than 10% of the storms (i.e., over 90% missing data rate). For this reason, a two-stage imputation approach will also be explored, filling originally missing values based on spatial interpolation, to enrich information for the implementation of the data completion-based imputation in the second stage. The specific data completion imputation method utilized here is probabilistic principal component analysis (PPCA) [33], as it can easily scale to large datasets, like the ones frequently encountered in storm surge imputation, and is well aligned with the PCA-based dimensionality reduction utilized when S_c and S_s metamodels are established based on GP surrogate modeling principles.

6.2. Imputation Using PPCA

PPCA-based imputation [33,56] establishes first a dimensionality reduction by projecting the original data (with missing values) onto its principal components (latent outputs) and subsequently estimates the missing values by recovering the data from its dimension-reduced form. The specific PPCA-based imputation variant utilized here follows the implementation adopted in [56]. A succinct description for it is provided next.

Formulation assumes a linear relationship between the original surge vector and the vector of principal components as the following:

$$z=Wt+\mu +\epsilon$$

(19)

where $t\in {ℝ}^{n_t}$ denotes the $n_t$-dimensional principal component vector, $W\in {ℝ}^{n_z\times n_t}$ the projection matrix, $\mu \in {ℝ}^{n_z}$ the mean vector with the ith component representing the mean of $z_i$ (mean of the ith column of $Z$), and $\epsilon \in {ℝ}^{n_z}$ the isotropic noise. Within the PPCA implementation, the probabilistic distributions for $t$ and $\epsilon$ are assumed, respectively, as $t~N(0,I)$ and $\epsilon ~N(0,{\sigma }^{2}I)$, where $N(m,\Sigma )$ represents the multivariate Gaussian distribution with the mean vector $m$ and covariance matrix $\Sigma$, $0$ the vector of zeros, $I$ the identity matrix, and ${\sigma }^2$ the magnitude of the error. These assumptions lead to the conditional distribution of the original surge vector $z$ given $t$ as the following:

$$z|t~N(Wt+\mu ,{\sigma }^{2}I).$$

(20)

Based on probability models in Equations (19) and (20), and using the surge response data (matrix Z) as samples of $z$, the complete-data log-likelihood of the n observations (correspond to the different storms) can be expressed as the following:

$$L={\displaystyle \sum _{h=1}^n\ln \{p(z^h,t^h)\}}={\displaystyle \sum _{h=1}^n[\ln \{p(z^h|t^h)\}+\ln \{p(t^h)\}]}$$

(21)

where $t^h=t(x^h)$ denotes the latent outputs for the hth storm. For maximizing this complete-data log-likelihood and, ultimately facilitating the data imputation, an Expectation-Maximization (EM)-based [57] iterative formulation is adopted for computational efficiency. The expectation step estimates the latent outputs using the observed portion of Z only, and then fills in missing entities in Z based on the estimated latent outputs (along with fixed $W$ and ${\sigma }^2$). The maximization step updates estimates for $W$ and ${\sigma }^2$ based on the updated complete matrix Z. Vector $\mu$ can be either updated at each iteration using imputed values or fixed to some reasonably initialized values at the beginning.

Following recommendations in [56], the adopted EM implementation also incorporates prior knowledge about expected patterns during the data completion process. This is critical for providing imputed data that exhibit similar patterns. Specifically, to the storm surge imputation problem, the pattern that we try to retain is to have the missing data in the storm surge database be below the node elevation (i.e., the pseudo-surge values corresponding to dry inundation state). For this reason, the missing entries in Z are initialized within the EM algorithm slightly (5 cm) below the corresponding node elevation.

6.3. Two-Stage Imputation Combining Spatial Interpolation and Data Completion Techniques

To enrich the information available for imputation relying on data completion, a two-stage formulation is promoted. In the first stage, spatial interpolation is first performed separately for each storm; for example, using kNN or GPI as detailed in Section 5. The incorrectly classified imputed data, corresponding to pseudo-surge values ${\underset{˜}{z}}_i^h>e_i$, are still considered to correspond to missing data. This leads to a partially imputed database that is subsequently imputed in the second stage using a data completion imputation technique. Through the introduction of the first stage, the database utilized in the data completion imputation is substantially enriched, while the enriched data are also guaranteed to facilitate correct classification based on the pseudo-surge.

6.4. Computational Complexity and Scalability of PPCA Imputation

Though the computational complexity of PPCA depends on the specifics of the EM algorithm used [58], with an appropriate EM implementation it can be reduced to be of order $O(h{n}_z{n}_{t}T)$, where T is the total number of EM iterations. This shows that, similar to kNN, PPCA may be considered to scale practically linearly with the size of the database (either the number of storms or nodes), showing that it can be easily extended to larger datasets. Compared to the computational complexity of the spatial interpolation techniques presented in Section 5.3, PPCA is expected to outperform GPI and have a similar computational burden as kNN. However, in general, PPCA outperforms even kNN if the progressive imputation of the latter requires a significant number of iterations.

7. Illustrative Case Study

The illustrative case study considers implementations of the proposed advances for establishing surrogate models for the CHS-LA database described in Section 3. The advantages of the different imputation advancements are examined with respect to the accuracy of the surrogate models that are subsequently calibrated using the imputed databases. The validation adopts a 10-fold CV implementation, following guidelines in Section 4.3.

7.1. Variants Examined

Results are presented for the entire database consisting of $n_z=$ 1,179,179 nodes, with $n_r=$ 488,216 being dry in at least one storm, as well as for a smaller domain around New Orleans with $n_z=$ 200,200 nodes, shown with the red boundary in Figure 3, with $n_r=$ 79,536 being dry in at least one storm, respectively. The smaller database is introduced in order to accommodate the efficient implementation of the GPI-based imputation. Even with adaptive covariance tapering, this implementation cannot scale to the $n_z=$ 1,179,179 nodes case unless a large-memory GPU-based workstation is utilized (see discussions in Section 5.3). A partitioning of the entire domain into smaller subdomains is needed to facilitate the GPI-based imputation. Without loss of generality, in the case study, the focus is placed on one of these subdomains only, with the understanding that implementation across the entire domain can be facilitated by repeating the process for other subdomains. The consideration of the subdomain allows, additionally, examination of the robustness of the different imputation strategies [especially of data-completion strategies as will be showcased in the discussions in Section 7.4] with respect to the size of the considered database. The notations D_f (full domain) and D_s (subdomain) are introduced to distinguish the different domains (and corresponding datasets). The spatial distribution of nodes, along with the proportion of instances in which they are inundated, is presented for D_f and D_s in Figure 3a and Figure 3b, respectively. Additionally, surge gaps of once-dry nodes within D_f and D_s are presented, respectively, in Figure 4a and Figure 4b. It is evident from these figures that the large storm surge gaps appear in regions with predominantly dry nodes, appearing next to flood protection systems around New Orleans. Note that Figure 3 and Figure 4 will be revisited in Section 7.5 when examining the spatial distribution of surrogate model prediction errors.

Imputation is performed using (i) spatial interpolation (kNN, GPI) or (ii) data completion (PPCA) as well as a combination of (i) and (ii) through the two-stage implementation presented in Section 6.3. Notations kNN, GPI, PPCA or, for the combined implementations, kNN-PPCA and GPI-PPCA will be used to distinguish the different cases. Note that the GPI variants (GPI and GPI-PPCA) will be only examined for the D_s domain. Additionally, for the kNN implementation, two formulations will be explored for incorporating hydraulic connectivity: the previous strategy [22] using the ADCIRC grid to define this connectivity and the newly proposed approach using a response-based connectivity. Notation kNN_rc will be used to differentiate the approach when the response-based connectivity is utilized, with notation kNN used for the original [22] formulation. The GPI implementation corresponds to sparsity ${\rho }_i=0.12\%$ [250 connected neighbor per node]. The surrogate models are established using either the pseudo-surge database or the corrected pseudo-surge database. Results are presented for the combination of S_s and S_c surrogate models for the pseudo-surge database and for the use of only the S_s surrogate model for the corrected pseudo-surge database, as the intention of the modification of the pseudo-surge within this database is to avoid the use of the secondary classification surrogate.

7.2. Imputation Results

Results are initially discussed for the imputation statistics. To better frame discussion, some statistics for the databases are first reviewed in

Table 2. Database statistics: proportion of nodes belonging to different groups as well as the respective percentages (%) of instances that these nodes are inundated within the original database.

Database	Statistics	Once-Dry	Surge Gap> (m)
			0.075	0.15	0.25	0.5	0.75	1	1.5
Df	% of nodes	41.40	21.77	13.91	8.86	3.77	1.86	0.99	0.35
	% inundated in database	31.26	6.94	4.40	2.90	1.94	1.88	2.04	2.89
Ds	% of nodes	39.73	37.18	27.67	20.26	10.59	5.99	3.55	1.28
	% inundated in database	42.54	4.75	3.18	2.27	1.46	1.21	1.16	1.63

. This table presents the proportion of the nodes belonging to different groups: the once-dry nodes as well as nodes with a surge gap over specific thresholds. The respective percentages (%) of instances, where these nodes are inundated within the original database, is also shown in this table.

Table 3. Imputation statistics for each of the individual imputation variants: proportion of nodes belonging to different groups as well as the respective percentages (%) of instances that these nodes are inundated within the original database.

Database	Statistics	kNN			kNNrc			PPCA			GPI
		${\mathit{A}}_{\mathit{p}\mathit{n}}$	${\mathit{C}}^2$	${\mathit{C}}^3$	${\mathit{A}}_{\mathit{p}\mathit{n}}$	${\mathit{C}}^2$	${\mathit{C}}^3$	${\mathit{A}}_{\mathit{p}\mathit{n}}$	${\mathit{C}}^2$	${\mathit{C}}^3$	${\mathit{A}}_{\mathit{p}\mathit{n}}$	${\mathit{C}}^2$	${\mathit{C}}^3$
Df	% of nodes	63.87	41.40	58.60	69.56	48.92	51.08	99.24	94.50	5.50	N/A
	% inundated in database	33.35	25.75	35.15	32.09	24.93	37.31	31.17	29.16	67.30
Ds	% of nodes	69.95	52.31	47.69	72.27	55.20	44.80	98.17	89.18	10.82	78.52	64.61	35.39
	% inundated in database	26.40	15.07	44.08	26.37	15.97	44.84	28.78	25.15	59.82	55.51	55.18	55.10

and

Table 4. Imputation statistics for the two-stage imputation variants: proportion of nodes belonging to different groups as well as the respective percentages (%) of instances that these nodes are inundated within the original database.

Database	Statistics	kNN-PPCA			kNNrc-PPCA			GPI-PPCA
		${\mathit{A}}_{\mathit{p}\mathit{n}}$	${\mathit{C}}^2$	${\mathit{C}}^3$	${\mathit{A}}_{\mathit{p}\mathit{n}}$	${\mathit{C}}^2$	${\mathit{C}}^3$	${\mathit{A}}_{\mathit{p}\mathit{n}}$	${\mathit{C}}^2$	${\mathit{C}}^3$
Df	% of nodes	54.58	41.00	59.00	60.45	46.98	53.02	N/A
	% inundated in database	33.34	26.81	34.34	32.13	26.23	35.70
Ds	% of nodes	63.49	52.85	47.15	65.55	55.20	44.80	73.09	59.27	40.73
	% inundated in database	24.49	16.77	42.50	25.08	17.74	42.66	55.55	55.10	55.23

then present similar statistics for the different imputation approaches, with Table 3 presenting results for the individual imputation variants, and Table 4 presenting results for the two-stage combination variants. Specifically, these tables present the proportion of the once-dry nodes that belong to the problematic node set $A_{pn}$, as well as the nodes that belong to groups $C^2$ and $C^3$. Recall that group $C^2$ corresponds to the modified problematic node set ${\overline{A}}_{pn}$ and $C^3$ to its complement. Similarly to Table 3, the respective percentages (%) of instances that these nodes are inundated within the original database is also shown.

Results show that the problematic nodes (group $A_{pn}$ or $C^2$) correspond to predominantly dry nodes in the original database, a trend agreeing with the one illustrated in [22]. Comparing across the imputation techniques relying on spatial interpolation (kNN variants and GPI), the kNN with response-based connectivity between nodes (kNN_rc) shows some small degradation of performance over kNN, with percentages of nodes belonging to groups $A_{pn}$ or $C^2$ increasing. GPI demonstrates an additional, larger performance degradation over the kNN variants. It should be pointed out that these preliminary results should not be interpreted as preference for kNN over kNN_rc and GPI; the performance of the imputation strategies needs to be examined with respect to the accuracy of the supported surrogate model and not with respect to small increases in the number of problematic nodes. On the other hand, the data completion strategy, PPCA, leads to a significant reduction in imputation quality, with almost all imputed nodes falling into the problematic category according to the results in Table 3. Though not definitive, this pattern provides a strong indication that this imputation approach will not be beneficial for peak-surge imputation.

Results for the two-stage combination variants in Table 4 show a significant improvement compared to PPCA, and moderate improvement for the kNN and GPI variants with respect to both $A_{pn}$ and $C^2$. For PPCA-based imputation, comparisons across Table 3 and Table 4 verify the expectations presented in Section 6.3. Enriching the information for predominantly dry nodes through initial spatial interpolation is highly beneficial for the data completion imputation strategies. The two-stage combination also illustrates improvements over the pure spatial interpolation strategies, especially GPI. To further examine the improvement achieved through the combination, the histograms of the misclassification of the imputation for the once-dry nodes, with (subplots in the left column) or without the data completion (subplots in the right column), are presented in Figure 5 and Figure 6, respectively, for D_f (full domain) and D_s (subdomain). In Figure 6, the results for kNN and kNN-PPCA are not presented as they are almost identical to those for kNN_rc and kNN_rc-PPCA. The results in the figures clearly demonstrate that the second stage of implementing PPCA-based data completion helps to reduce the portion of nodes with especially high misclassification (close to or over 80%), which is consistent with the observations from Table 3 and Table 4. As discussed in the previous paragraph, the important question is whether these benefits translate to improved predictive accuracy for the surrogate model developed based on the imputed datasets. It is important to note that the trends across all tables are very comparable between the domains D_f and D_s, illustrating a degree of similarity between them.

7.3. Surrogate Model Performance for the Entire Domain

The performance of the imputation strategies is evaluated next with respect to the accuracy of the established surrogate models. In this section, results for the entire database D_f are discussed, meaning that only kNN variants are examined for imputation using spatial interpolation. Results for the subdomain D_s will be examined in the next section. The validation metrics correspond to the misclassification and surge score discussed in Section 4.3. Results are presented for different node groupings: the entire domain, the once-dry nodes, nodes set for different surge gaps, and groups $C^2$ and $C^3$. It should be pointed out that groups $C^2$ and $C^3$ change across the variants, as illustrated earlier in Table 3 and Table 4, meaning that comparisons of their statistics across the variants do not yield any meaningful trends. Results for these groups are only presented for examining trends within each variant.

Table 5. Surge score $\overline{SC}$ (cm) for different nodal groups for full domain (D_f), for surrogate models supported by different imputation variants. The best accuracy across different imputation variants is highlighted in bold.

		Pseudo-Surge Database (Combination of Ss and Sc)					Corrected Pseudo-Surge Database (Ss Only)
		kNN	kNNrc	PPCA	kNN-PPCA	kNNrc-PPCA	kNN	kNNrc	PPCA	kNN-PPCA	kNNrc-PPCA
All nodes		7.69	7.69	7.79	7.51	7.53	7.85	7.87	8.10	7.69	7.73
Once-dry		5.32	5.27	5.40	5.13	5.12	5.55	5.60	5.83	5.42	5.49
Surge gap> (m)	0.25	2.27	2.18	2.11	2.17	2.10	2.88	3.13	2.94	2.79	3.05
	0.5	1.85	1.75	1.74	1.81	1.72	2.52	2.91	2.59	2.44	2.83
	0.75	1.76	1.68	1.71	1.76	1.69	2.58	3.10	2.64	2.47	3.02
	1	1.89	1.81	1.86	1.90	1.84	2.90	3.47	2.93	2.77	3.35
	1.5	2.67	2.56	2.72	2.75	2.70	4.38	4.83	4.40	4.24	4.66
Node groups	$C^2$	5.76	5.41	5.29	5.62	5.43	6.21	6.00	5.74	6.24	6.14
	$C^3$	5.00	5.13	7.29	4.79	4.85	5.08	5.22	7.45	4.86	4.92

shows results for the surge score $\overline{SC}$,

Table 6. Misclassification $\overline{MC}$ (%) for different nodal groups for full domain (D_f), for surrogate models supported by different imputation variants. The best accuracy across different imputation variants is highlighted in bold.

		Pseudo-Surge Database (Combination of Ss and Sc)					Corrected Pseudo-Surge Database (Ss Only)
		kNN	kNNrc	PPCA	kNN-PPCA	kNNrc-PPCA	kNN	kNNrc	PPCA	kNN-PPCA	kNNrc-PPCA
All nodes		1.54	1.55	1.68	1.53	1.54	2.60	2.79	3.53	2.46	2.62
Once-dry		3.57	3.57	3.90	3.54	3.55	6.14	6.58	8.38	5.79	6.18
Surge gap> (m)	0.25	1.01	1.00	1.02	1.00	0.99	5.04	7.15	6.11	4.38	6.01
	0.5	0.77	0.76	0.77	0.76	0.75	4.60	7.72	5.52	4.00	6.41
	0.75	0.69	0.68	0.69	0.69	0.68	4.55	8.53	5.24	4.00	7.03
	1	0.70	0.68	0.69	0.69	0.68	4.59	8.65	5.17	4.14	6.96
	1.5	0.86	0.84	0.84	0.86	0.84	5.24	7.96	6.32	5.25	6.63
Node groups	$C^2$	3.63	3.67	4.03	3.72	3.75	8.37	8.86	8.69	7.83	8.38
	$C^3$	3.52	3.47	1.67	3.42	3.37	4.56	4.39	2.94	4.38	4.24

results for the total misclassification $\overline{MC}$,

Table 7. False positive misclassification ${}^{+}\overline{MC}$ (%) for different nodal groups for full domain (D_f), for surrogate models supported by different imputation variants. The best accuracy across different imputation variants is highlighted in bold.

		Pseudo-Surge Database (Combination of Ss and Sc)					Corrected Pseudo-Surge Database (Ss Only)
		kNN	kNNrc	PPCA	kNN-PPCA	kNNrc-PPCA	kNN	kNNrc	PPCA	kNN-PPCA	kNNrc-PPCA
All nodes		0.68	0.67	0.62	0.66	0.66	1.82	2.05	2.90	1.68	1.89
Once-dry		1.63	1.61	1.50	1.60	1.59	4.39	4.94	7.01	4.07	4.56
Surge gap> (m)	0.25	0.31	0.30	0.32	0.30	0.30	4.82	7.00	6.02	4.15	5.84
	0.5	0.21	0.20	0.21	0.21	0.20	4.41	7.60	5.43	3.81	6.29
	0.75	0.19	0.18	0.19	0.19	0.18	4.38	8.42	5.15	3.83	6.91
	1	0.22	0.20	0.20	0.22	0.20	4.43	8.52	5.06	3.99	6.82
	1.5	0.34	0.28	0.28	0.34	0.28	5.12	7.85	6.22	5.13	6.48
Node groups	$C^2$	1.36	1.39	1.52	1.37	1.40	7.22	7.78	7.34	6.69	7.28
	$C^3$	1.83	1.82	1.12	1.76	1.75	2.40	2.23	1.24	2.24	2.15

for the false positive misclassification ${}^{+}\overline{MC}$, and

Table 8. False negative misclassification for ${}^{-}\overline{MC}$ (%) for different nodal groups for full domain (D_f), for surrogate models supported by different imputation variants. The best accuracy across different imputation variants is highlighted in bold.

		Pseudo-Surge Database (Combination of Ss and Sc)					Corrected Pseudo-Surge Database (Ss Only)
		kNN	kNNrc	PPCA	kNN-PPCA	kNNrc-PPCA	kNN	kNNrc	PPCA	kNN-PPCA	kNNrc-PPCA
All nodes		0.86	0.88	1.06	0.87	0.88	0.78	0.74	0.63	0.78	0.73
Once-dry		1.93	1.96	2.41	1.94	1.97	1.74	1.63	1.37	1.73	1.62
Surge gap> (m)	0.25	0.70	0.70	0.70	0.70	0.69	0.22	0.15	0.09	0.24	0.16
	0.5	0.55	0.55	0.56	0.55	0.55	0.19	0.12	0.08	0.19	0.13
	0.75	0.50	0.50	0.50	0.49	0.50	0.17	0.11	0.09	0.17	0.12
	1	0.47	0.49	0.49	0.47	0.49	0.16	0.12	0.11	0.15	0.14
	1.5	0.52	0.56	0.56	0.52	0.56	0.12	0.11	0.10	0.12	0.14
Node groups	$C^2$	2.27	2.28	2.51	2.34	2.35	1.15	1.09	1.35	1.14	1.09
	$C^3$	1.69	1.66	0.54	1.67	1.62	2.16	2.15	1.71	2.13	2.09

for the false negative ${}^{-}\overline{MC}$ misclassification. As discussed in Section 7.1, results are presented for the combination of S_s and S_c surrogate models for the pseudo-surge database and for the use of only the S_s surrogate model for the corrected pseudo-surge database. To examine in detail the imputation quality, corresponding results when using the S_s surrogate model for the pseudo-surge database are presented in

Table A1. Surge score $\overline{SC}$ (cm) for different node groups for D_f domain for surrogate models (S_s only) based on pseudo-surge database imputed by different variants.

		Pseudo-Surge Database (Ss Only)
		kNN	kNNrc	PPCA	kNN-PPCA	kNNrc-PPCA
All nodes		13.39	13.90	9.25	7.91	7.98
Once-dry		19.08	20.28	8.93	6.08	6.22
Surge gap> (m)	0.25	59.78	72.37	5.39	4.11	4.90
	0.5	70.61	88.33	4.68	3.64	4.85
	0.75	87.09	106.73	4.72	3.73	5.30
	1	109.14	122.55	5.26	4.21	5.82
	1.5	139.46	142.92	7.95	6.43	7.67
Node groups	$C^2$	38.95	36.06	9.03	7.88	7.73
	$C^3$	5.04	5.16	7.30	4.82	4.88

,

Table A2. Misclassification $\overline{MC}$ (%) for different node groups from D_f for surrogate models (S_s only) based on pseudo-surge database imputed by different variants.

		Pseudo-Surge Database (Ss Only)
		kNN	kNNrc	PPCA	kNN-PPCA	kNNrc-PPCA
All nodes		7.19	7.68	10.82	3.70	4.17
Once-dry		17.22	18.38	25.99	8.79	9.91
Surge gap> (m)	0.25	46.94	47.91	19.90	11.68	15.86
	0.5	53.76	55.55	18.12	11.34	17.78
	0.75	63.20	65.39	17.94	12.09	20.34
	1	71.31	74.12	18.81	12.99	21.25
	1.5	72.73	78.47	23.42	15.48	21.87
Node groups	$C^2$	35.15	32.99	27.33	15.15	16.33
	$C^3$	4.55	4.38	2.97	4.36	4.23

,

Table A3. False positive misclassification ${}^{*}\overline{MC}$ (%) for different node groups from D_f for surrogate models (S_s only) based on pseudo-surge database imputed by different variants.

		Pseudo-Surge Database (Ss Only)
		kNN	kNNrc	PPCA	kNN-PPCA	kNNrc-PPCA
All nodes		6.46	7.00	10.36	2.96	3.47
Once-dry		15.61	16.90	25.02	7.14	8.39
Surge gap> (m)	0.25	46.79	47.81	19.85	11.46	15.71
	0.5	53.65	55.49	18.06	11.16	17.67
	0.75	63.12	65.34	17.87	11.93	20.23
	1	71.26	74.09	18.73	12.85	21.12
	1.5	72.73	78.47	23.33	15.37	21.74
Node groups	$C^2$	34.28	32.18	26.40	14.18	15.41
	$C^3$	2.41	2.26	1.36	2.25	2.16

and

Table A4. False negative misclassification ${}^{-}\overline{MC}$ (%) for different node groups from D_f for surrogate models (S_s only) based on pseudo-surge database imputed by different variants.

		Pseudo-Surge Database (Ss Only)
		kNN	kNNrc	PPCA	kNN-PPCA	kNNrc-PPCA
All nodes		0.73	0.68	0.46	0.74	0.70
Once-dry		1.61	1.48	0.97	1.65	1.53
Surge gap> (m)	0.25	0.15	0.10	0.06	0.22	0.15
	0.5	0.11	0.06	0.06	0.18	0.11
	0.75	0.08	0.05	0.07	0.16	0.12
	1	0.05	0.03	0.08	0.14	0.13
	1.5	0.01	0.01	0.09	0.11	0.13
Node groups	$C^2$	0.87	0.81	0.93	0.97	0.91
	$C^3$	2.14	2.13	1.61	2.12	2.07

in Appendix G. The presentation mirrors the ones in Table 5, Table 6, Table 7 and Table 8. It should be pointed out that this formulation is not a recommended implementation for storm surge surrogate modeling [22]; the two formulations whose results are presented in Table 5, Table 6, Table 7 and Table 8 are the promoted alternatives.

Before discussing the results, it is important to stress that to evaluate and ultimately compare the performance of the imputation strategies, all the tables need to be comprehensively examined altogether, with primary focus on the surge score (Table 5) which directly represents the predictive accuracy of the resultant surrogate models, and secondary focus on the total misclassification (Table 6) which quantifies the quality of the inundation state predictions. The false positive and false negative misclassification results can be used to interpret the trends in the performance and should be linked to the surge score and total misclassification, and to the characteristics of each imputation strategy, as discussed in Section 5 or Section 6. For example, a low false negative rate that is not combined with a low total misclassification rate does not show superiority, rather that there is an over prediction bias. Such a bias will be naturally reflected with a higher surge score and therefore will be identified when results across all tables are comprehensively evaluated.

Performance across the different databases or combination of regression and classification options verify the trends observed in [22]. The combination of S_s and S_c provides the best overall performance with results relying on the use of S_s only consistently yielding lower accuracy (larger surge score values in Table 5 and Table A1 and larger misclassification in Table 6 and Table A2), predominantly originating from false positive predictions as evident from Table 7and Table A3. The performance deteriorates significantly when only S_s is implemented utilizing the pseudo-surge database without any corrections (Table A1, Table A2, Table A3 and Table A4). As discussed in Section 4, these trends, especially the propensity for false positives, are expected and originate from the wrong information introduced into the database during the imputation process supporting the development of the regression surrogate model. Note that this propensity still exists when the corrected pseudo-surge database is utilized (Table 5, Table 6, Table 7 and Table 8). The propensity, in this case, can be attributed [22] to the inability of the correction (setting surge value to be always smaller than the node elevation) to capture the appropriate underlying variation characteristics for the pseudo-surge across the storms with different characteristics in the original database. However, the predictive accuracy of all considered surrogate model implementations deteriorates consistently for the problematic nodes corresponding to large surge gaps, demonstrating the challenges associated with this type of nodes.

Moving now to the comparison across the different imputation variants, focusing first on the different kNN strategies, both exhibit similarly good performance in general (Table 5, Table 6, Table 7 and Table 8). However, the use of the response-based connectivity between nodes, i.e., the kNN_rc case, results in the best performance (lower surge score and best misclassification results) for the surrogate model implementation corresponding to the combination of S_s and S_c, especially for the problematic nodes with larger surge gaps (Table 5). These are important results, as they demonstrate that incorporating hydraulic connectivity based on surge response similarity into the imputed database can improve the predictive performance of the surrogate model for more challenging regions without compromising its accuracy for other regions, which was the motivation for introducing the new concept of hydraulic connectivity. When predictions rely strictly on the use of the regression surrogate model (S_s), either utilizing the pseudo-surge database or the corrected pseudo-surge database, kNN demonstrates superior performance compared to kNN_rc, primarily originating from better false positive misclassification accuracy (Table 7 and Table A3). This pattern agrees with the trends reported earlier in Table 3 related to kNN yielding smaller percentage of problematic nodes. In this case, however, these trends suggest that while S_s developed from the kNN-imputed database tends to underestimate surge levels for nodes with large surge gaps, the higher false positive rate of S_s developed from the kNN_rc-imputed database can actually improve final prediction accuracy for these nodes when combined with S_c. This stresses, again, an advantage for introducing response-based connectivity information in the imputation process, as the approach can ultimately promote more accurate predictions for the storm surge.

Imputation based entirely on data completion (PPCA) shows superior performance compared to the other imputation strategies when considering the implementation that uses S_s only, based on the pseudo-surge database (Table A1, Table A2, Table A3 and Table A4). Its superiority originates from better false positive misclassification for the highly problematic nodes with larger surge gaps. This should be attributed to the ability of the promoted data completion strategy to better fit the underlying trends for the original database. Still, this performance is substantially worse compared to strategies relying on the combination of S_s and S_c surrogates or the use of the corrected pseudo-surge database (Table 5, Table 6, Table 7 and Table 8), and so it should not be interpreted as superiority of the PPCA imputation strategy. In fact, in Table 5, Table 6, Table 7 and Table 8, PPCA slightly underperforms the spatial interpolation variants, with vulnerabilities originating from the false positive misclassification for the corrected pseudo-surge database strategy (use of S_s only) and from the false negative misclassification for the pseudo-surge database strategy (use of combination of S_s and S_c). This change in the patterns should be attributed to the larger portion of nodes identified as problematic for the PPCA implementation (Table 2). Overall, the discussions indicate a small preference for the spatial imputation strategies over the pure data completion ones.

Finally, examining the two-stage imputation strategies, kNN-PPCA and kNN_rc-PPCA, they demonstrate the best overall performance and the most robust trends across the different surrogate modeling strategies. This is not surprising, as they combine the two imputation approaches to fully leverage their complementary strengths and include an intermediate check that ultimately reduces the number of problematic nodes, as evident in the results in Table 4 (compare values to those in Table 3). This intermediate check and the enrichment of the information for PPCA contributes to a remarkable improvement, compared to the alternative imputation variants, when considering implementation using the S_s only, based on the pseudo-surge database (Table A1, Table A2, Table A3 and Table A4). More importantly, it yields some improvements, though admittedly smaller compared to the other imputation variants, for the promoted surrogate modeling alternatives presented in Table 5, Table 6, Table 7 and Table 8. The improved performance seems to originate from better misclassification rates. Overall the discussion shows that for peak storm surge predictions, the combination of spatial interpolation (stage 1) and data completion (stage 2) imputation strategies supports higher accuracy surrogate modeling predictions, a trend that agrees with the results reported in [26] for time-series predictions.

7.4. Surrogate Model Performance for the Subdomain

This section evaluates the performance of the imputation strategies with respect to the accuracy of the established surrogate models for the subdomain D_s. Results are presented for the two promoted surrogate model alternative implementations in

Table 9. Surge score $\overline{SC}$ (cm) for different node groups for subdomain (D_s), for surrogate models supported by different imputation variants. The best accuracy across different imputation variants is highlighted in bold.

		Pseudo-Surge Database (Combination of Ss and Sc)							Corrected Pseudo-Surge Database (Ss Only)
		kNN	kNNrc	GPI	PPCA	kNN-PPCA	kNNrc-PPCA	GPI-PPCA	kNN	kNNrc	GPI	PPCA	kNN-PPCA	kNNrc-PPCA	GPI-PPCA
All nodes		9.26	9.25	9.13	9.63	9.39	9.40	9.20	9.61	9.67	10.72	10.08	9.58	9.83	9.91
Once-dry		5.91	5.82	5.85	6.06	5.77	5.69	5.64	6.48	6.62	7.29	6.78	6.25	6.52	7.43
Surge gap> (m)	0.25	1.98	1.86	1.86	1.84	1.92	1.81	1.84	2.73	3.34	2.82	2.70	2.60	3.24	3.92
	0.5	1.58	1.46	1.47	1.47	1.55	1.46	1.49	2.31	3.06	2.35	2.28	2.20	2.97	3.37
	0.75	1.36	1.29	1.30	1.32	1.36	1.30	1.33	2.11	2.97	2.15	2.09	2.01	2.91	3.11
	1	1.35	1.28	1.29	1.33	1.35	1.31	1.34	2.13	2.93	2.22	2.15	2.03	2.86	3.17
	1.5	1.98	1.87	1.89	1.99	2.03	1.98	2.01	3.15	3.61	3.40	3.29	3.12	3.57	5.09
Node groups	$C^2$	5.01	4.99	5.62	6.00	4.95	4.94	5.22	5.91	6.30	6.70	6.81	5.95	6.41	8.22
	$C^3$	6.90	6.84	6.38	6.48	6.56	6.52	6.26	7.09	7.02	8.37	6.57	6.53	6.64	6.27

,

Table 10. Misclassification $\overline{MC}$ (%) for different node groups for subdomain (D_s), for surrogate models supported by different imputation variants. The best accuracy across different imputation variants is highlighted in bold.

		Pseudo-Surge Database (Combination of Ss and Sc)							Corrected Pseudo-Surge Database (Ss Only)
		kNN	kNNrc	GPI	PPCA	kNN-PPCA	kNNrc-PPCA	GPI-PPCA	kNN	kNNrc	GPI	PPCA	kNN-PPCA	kNNrc-PPCA	GPI-PPCA
All nodes		1.13	1.13	1.15	1.21	1.12	1.13	1.14	2.37	2.97	3.78	3.35	2.21	2.78	4.40
Once-dry		2.78	2.76	2.80	2.96	2.75	2.74	2.79	5.90	7.40	9.40	8.34	5.49	6.90	11.00
Surge gap> (m)	0.25	0.78	0.78	0.77	0.78	0.78	0.77	0.78	5.47	10.37	4.83	5.03	4.80	9.21	11.19
	0.5	0.60	0.59	0.59	0.59	0.60	0.59	0.59	5.21	11.11	4.03	4.38	4.53	9.93	10.33
	0.75	0.51	0.51	0.51	0.51	0.51	0.51	0.51	5.02	11.51	3.55	3.99	4.40	10.43	9.81
	1	0.51	0.51	0.51	0.51	0.51	0.50	0.51	4.83	10.80	3.61	4.11	4.41	9.57	9.94
	1.5	0.67	0.65	0.66	0.65	0.66	0.65	0.66	5.20	8.41	4.77	5.48	5.28	6.92	13.74
Node groups	$C^2$	2.15	2.25	2.63	3.21	2.22	2.29	2.70	7.46	10.20	9.00	9.33	7.23	9.77	16.27
	$C^3$	3.47	3.38	3.11	1.28	3.26	3.23	2.92	4.18	3.95	10.15	1.90	3.81	3.74	3.32

,

Table 11. False positive misclassification ${}^{+}\overline{MC}$ (%) for different node groups for subdomain (D_s), for surrogate models supported by different imputation variants. The best accuracy across different imputation variants is highlighted in bold.

		Pseudo-Surge Database (Combination of Ss and Sc)							Corrected Pseudo-Surge Database (Ss Only)
		kNN	kNNrc	GPI	PPCA	kNN-PPCA	kNNrc-PPCA	GPI-PPCA	kNN	kNNrc	GPI	PPCA	kNN-PPCA	kNNrc-PPCA	GPI-PPCA
All nodes		0.50	0.50	0.49	0.51	0.49	0.50	0.48	1.82	2.47	3.49	3.07	1.68	2.28	3.97
Once-dry		1.26	1.26	1.22	1.28	1.25	1.25	1.22	4.57	6.22	8.77	7.72	4.23	5.75	10.00
Surge gap> (m)	0.25	0.23	0.23	0.23	0.24	0.23	0.23	0.24	5.19	10.19	4.62	4.87	4.54	9.04	11.07
	0.5	0.17	0.16	0.16	0.16	0.17	0.16	0.16	4.98	10.95	3.84	4.23	4.30	9.79	10.23
	0.75	0.14	0.13	0.14	0.13	0.14	0.13	0.14	4.83	11.37	3.37	3.84	4.21	10.29	9.71
	1	0.14	0.14	0.14	0.14	0.14	0.14	0.14	4.65	10.65	3.45	3.97	4.22	9.43	9.85
	1.5	0.22	0.20	0.21	0.20	0.21	0.20	0.20	5.05	8.27	4.63	5.35	5.13	6.78	13.66
Node groups	$C^2$	0.76	0.84	1.06	1.34	0.78	0.85	1.05	6.79	9.60	8.39	8.74	6.58	9.21	15.77
	$C^3$	1.80	1.79	1.52	0.87	1.69	1.69	1.46	2.13	2.06	9.48	1.02	1.96	1.94	1.60

and

Table 12. False negative misclassification ${}^{-}\overline{MC}$ (%) for different node groups for subdomain (D_s), for surrogate models supported by different imputation variants. The best accuracy across different imputation variants is highlighted in bold.

		Pseudo-Surge Database (Combination of Ss and Sc)							Corrected Pseudo-Surge Database (Ss Only)
		kNN	kNNrc	GPI	PPCA	kNN-PPCA	kNNrc-PPCA	GPI-PPCA	kNN	kNNrc	GPI	PPCA	kNN-PPCA	kNNrc-PPCA	GPI-PPCA
All nodes		0.63	0.63	0.66	0.70	0.63	0.63	0.66	0.56	0.50	0.30	0.28	0.53	0.49	0.43
Once-dry		1.52	1.50	1.58	1.50	1.50	1.49	1.57	1.32	1.18	0.63	0.62	1.26	1.15	1.00
Surge gap> (m)	0.25	0.55	0.54	0.54	0.54	0.54	0.54	0.54	0.28	0.19	0.20	0.16	0.26	0.17	0.12
	0.5	0.43	0.43	0.43	0.43	0.43	0.43	0.43	0.23	0.16	0.19	0.16	0.22	0.14	0.11
	0.75	0.38	0.38	0.38	0.38	0.38	0.37	0.38	0.20	0.14	0.18	0.15	0.19	0.14	0.10
	1	0.37	0.37	0.37	0.37	0.37	0.36	0.37	0.18	0.14	0.17	0.14	0.18	0.14	0.10
	1.5	0.45	0.45	0.45	0.45	0.45	0.45	0.45	0.15	0.14	0.14	0.12	0.16	0.14	0.09
Node groups	$C^2$	1.38	1.42	1.57	1.87	1.44	1.44	1.64	0.67	0.60	0.61	0.59	0.64	0.57	0.51
	$C^3$	1.67	1.59	1.58	0.41	1.57	1.54	1.45	2.05	1.89	0.67	0.88	1.85	1.79	1.72

, mirroring the presentation format of Table 5, Table 6, Table 7 and Table 8. Additionally, results are presented in

Table A5. Surge score $\overline{SC}$ (cm) for different node groups from D_s for surrogate models (S_s only) based on pseudo-surge database imputed by different variants.

		Pseudo-Surge Database (Ss Only)
		kNN	kNNrc	GPI	PPCA	kNN-PPCA	kNNrc-PPCA	GPI-PPCA
All nodes		26.31	31.17	29.54	11.11	9.96	10.13	9.91
Once-dry		48.82	61.01	57.15	9.78	7.22	7.54	7.43
Surge gap> (m)	0.25	108.97	151.33	129.24	4.73	3.75	5.04	3.92
	0.5	118.19	164.75	140.34	3.96	3.21	4.78	3.37
	0.75	131.17	177.32	153.55	3.62	2.98	4.76	3.11
	1	145.46	178.84	165.69	3.73	3.01	4.60	3.17
	1.5	177.34	190.90	195.68	5.88	4.66	5.42	5.09
Node groups	$C^2$	87.01	104.95	84.83	10.28	7.86	8.45	8.22
	$C^3$	6.93	6.87	6.64	6.49	6.60	6.55	6.27

,

Table A6. Misclassification $\overline{MC}$ (%) for different node groups from D_s for surrogate models (S_s only) based on pseudo-surge database imputed by different variants.

		Pseudo-Surge Database (Ss Only)
		kNN	kNNrc	GPI	PPCA	kNN-PPCA	kNNrc-PPCA	GPI-PPCA
All nodes		13.83	13.55	15.01	9.49	3.77	4.77	4.40
Once-dry		34.73	34.03	37.68	23.81	9.41	11.90	11.00
Surge gap> (m)	0.25	76.92	78.04	79.88	15.58	11.29	19.14	11.19
	0.5	83.44	85.15	86.48	13.86	10.87	20.20	10.33
	0.75	87.79	89.20	89.31	12.81	10.76	20.51	9.81
	1	88.66	89.75	89.41	13.34	11.05	18.59	9.94
	1.5	87.58	88.15	88.15	18.53	14.16	15.26	13.74
Node groups	$C^2$	62.61	58.46	56.41	27.14	15.18	19.35	16.27
	$C^3$	4.14	3.93	3.51	2.00	3.83	3.70	3.32

,

Table A7. False positive misclassification ${}^{+}\overline{MC}$ (%) for different node groups from D_s for surrogate models (S_s only) based on pseudo-surge database imputed by different variants.

		Pseudo-Surge Database (Ss Only)
		kNN	kNNrc	GPI	PPCA	kNN-PPCA	kNNrc-PPCA	GPI-PPCA
All nodes		13.32	13.09	14.60	9.29	3.25	4.29	3.97
Once-dry		33.52	32.95	36.75	23.39	8.19	10.79	10.00
Surge gap> (m)	0.25	76.79	77.95	79.84	15.50	11.08	18.99	11.07
	0.5	83.35	85.08	86.47	13.78	10.69	20.07	10.23
	0.75	87.74	89.15	89.30	12.72	10.61	20.38	9.71
	1	88.63	89.73	89.41	13.25	10.91	18.46	9.85
	1.5	87.57	88.15	88.15	18.45	14.04	15.13	13.66
Node groups	$C^2$	62.14	58.02	56.00	26.78	14.62	18.85	15.77
	$C^3$	2.13	2.07	1.63	1.15	1.97	1.93	1.60

and

Table A8. False negative misclassification ${}^{-}\overline{MC}$ (%) for different node groups from D_s for surrogate models (S_s only) based on pseudo-surge database imputed by different variants.

		Pseudo-Surge Database (Ss Only)
		kNN	kNNrc	GPI	PPCA	kNN-PPCA	kNNrc-PPCA	GPI-PPCA
All nodes		0.51	0.46	0.41	0.20	0.52	0.48	0.43
Once-dry		1.21	1.08	0.93	0.42	1.22	1.11	1.00
Surge gap> (m)	0.25	0.13	0.09	0.04	0.08	0.21	0.15	0.12
	0.5	0.09	0.07	0.01	0.08	0.18	0.13	0.11
	0.75	0.05	0.05	0.01	0.09	0.16	0.13	0.10
	1	0.03	0.03	0.00	0.09	0.14	0.13	0.10
	1.5	0.01	0.01	0.00	0.09	0.12	0.13	0.09
Node groups	$C^2$	0.48	0.45	0.41	0.36	0.56	0.50	0.51
	$C^3$	2.00	1.86	1.88	0.85	1.85	1.77	1.72

for the implementation using only the S_s surrogate model with the pseudo-surge database, mirroring the presentation format of Table A1, Table A2, Table A3 and Table A4. In all instances, results for the GPI imputation strategy are also reported in this section.

With respect to the remaining imputation strategies, excluding the GPI implementation, the trends are similar to the ones presented in Section 7.3, with the primary difference being a relatively small reduction in the performance of the imputation strategies utilizing data completion, especially PPCA, but also kNN-PPCA and kNN_rc-PPCA. This should be attributed to the 6-fold reduction in the database size (from D_f to D_s). This negatively influences the quality of imputation by data completion, and subsequently the performance of the established regression surrogate model. The consistency of the remaining trends, and the justified small reduction in the performance of the approaches relying on data completion imputation, provide a strong validation of the results and arguments presented in Section 7.3, demonstrating that they are not database dependent.

Focusing now on the performance of GPI imputation, it offers comparable results to the other imputation approaches relying on spatial interpolation, kNN and kNN_rc, demonstrating that it is a viable alternative. This holds true both on its own (GPI) and when combined with data completion within the two-stage imputation strategy (GPI-PPCA). The trends vary across the different surrogate model strategies, i.e., whether the combination of S_s and S_c is used or whether predictions are established using solely S_s based on the pseudo-surge database or the corrected pseudo-surge database. The best overall performance is established when utilizing the combination of S_s and S_c, as expected, and performing imputation based on GPI or GPI-PPCA. For the highly problematic nodes, corresponding to larger surge gaps, kNN_rc still remains the preferred choice.

Overall, these comparisons, coupled with the discussions in Section 7.3, demonstrate that spatial interpolation with global characteristics (i.e., GPI) tends to better capture the global trends for the surge behavior compared to spatial interpolation based on localized characteristics (kNN variants). Still, the latter have the ability to better capture the discontinuous behavior around problematic nodes corresponding to large surge gaps, especially when connectivity information based on flooding patterns is leveraged within the imputation process, as evident by the superiority of kNN_rc over kNN. Combining spatial interpolation with data completion imputation strategies offers additional robustness in the predictions and has the potential to further improve accuracy, depending on the quality of the information (size of database) available for the data completion approaches.

7.5. Trends for Distribution of Errors

This section examines the distribution of errors (surge score and misclassification) for the best performing variants that were identified in Section 7.3 and Section 7.4. These correspond to kNN_rc and GPI as well as their counterparts when coupled with PPCA, kNN_rc-PPCA, and GPI-PPCA (all variants corresponding to those using pseudo-surge database with surrogate model combination of S_s and S_c). Figure 7 and Figure 8 present results for the spatial distribution of the surge score across the nodes, with Figure 7 offering results for the entire domain D_f and Figure 8 for the subdomain D_s. Figure 9 and Figure 10 present results for the histogram of misclassification for the same implementation variants illustrated in Figure 7 and Figure 8, corresponding to the entire domain D_f and Figure 8 for the subdomain D_s, respectively.

Focusing first on the spatial distributions of the surge score (Figure 7 and Figure 8), the surge scores of nodes that are closer to those with large surge gaps are generally larger (see the locations of once-dry nodes and their surge gaps in Figure 4), whereas predominantly dry nodes (those with lower wetness in Figure 3) have small surge score values regardless of their proximity to the nodes with large surge gaps. The former trend is expected as nodes with larger surge gaps pose greater challenges in achieving accurate surge predictions, both for themselves as well as for nearby nodes. The better accuracy for the predominantly dry nodes can be attributed to the incorporation of the classification surrogate model S_c, which enhances the predictive accuracy of node conditions significantly. When comparing surge score distributions between variants, the spatial distribution trends remain largely the same, with very small differences across the different variants. This is true even when focusing on the comparisons to the variants with (subplots in the right column) and without data-completion (subplots in the left column); the two-stage imputation strategy contributes to a slight general improvement in the accuracy, consistent with observations reported from Table 5 and Table 9, though the distributions do not significantly change. The patterns are the same for both the entire domain D_f and the subdomain D_s.

Examining next the histograms of the misclassification (Figure 9 and Figure 10), small misclassification is reported for most of the domain, with nodes that exhibit higher than 5% misclassification corresponding to only a small portion of the database: less than 15% for the entire domain D_f and less than 10% for the subdomain D_s. This result shows a remarkable robustness with respect to appropriately classifying the correct inundation state (dry or wet) for the vast majority of the nodes within the domain of interest. Again, small differences are exhibited across the variants, verifying the trends exhibited in Figure 7 and Figure 8 for similarity of the predictive performance across the nodes within the broader domain. This shows that the different variant implementations promoted here share the same robustness when evaluated with respect to the worst-case performance for individual nodes within the domain, either with respect to surge score (Figure 7 and Figure 8) or misclassification (Figure 9 and Figure 10) of the node inundation state.

8. Conclusions

The development of surrogate models or metamodels for the prediction of peak storm surge requires imputation of the original simulation data for inland nodes that have remained dry in some of the synthetic storm simulations to estimate the so-called pseudo-surge. This paper examined three distinct advancements for the imputation of peak storm surge databases. In the first advancement, the use of response-based correlation was examined for infusing hydraulic connectivity information within a kNN based spatial interpolation. The response-based correlation between nodes was defined using the binary response information representing the condition of each node (i.e., inundated or not), with only nodes whose correlation with the target node is higher than a specific threshold considered as candidate neighbors in the kNN implementation. In the second advancement, a Gaussian Process interpolation (GPI) was examined as a spatial imputation strategy, relying on a recently established adaptive covariance tapering scheme for accommodating an efficient GPI implementation for large datasets. In the third advancement, imputation using data completion, specifically utilizing PPCA, was examined as an alternative to spatial interpolation techniques. Additionally, the combination of spatial imputation with PPCA was also considered, with the objective to enrich the information before performing PPCA. The advantages of the advancements are examined with respect to the accuracy of the surrogate model that is subsequently calibrated using the imputed databases.

An illustrative case study regarding surrogate modeling predictions for Louisiana, focusing on New Orleans, was considered utilizing (and imputing) the Coastal Hazards System-Louisiana (CHS-LA) coastal storm hazards study database. A comprehensive k-fold cross validation implementation was established, and the key outcomes of the investigation are as follows:

• As reported in previous studies, the best overall performance can be established by combining a regression and a classification surrogate models, with the regression model developed utilizing directly the imputed database without any corrections.
• Spatial interpolation using response-based connectivity information can better capture discontinuous patterns in surge responses, demonstrating superiority in supporting surrogate model development with improved predictive accuracy at problematic nodes. Note that such nodes can be identified by the larger surge gaps in the original data.
• Imputation utilizing GPI is a viable alternative (compared to kNN) imputation strategy based on spatial interpolation, and can better capture the global surge trends, offering superior overall performance. However, it cannot capture discontinuous patterns in surge responses, and extending GPI using response-based connectivity information within the covariance kernel is not straightforward. Nevertheless, this is a topic that should be investigated in the future.
• Data completion imputation for the peak-surge demonstrates very poor performance, originating from the fact that for many (predominantly dry) nodes, very limited information is available in the original database to establish the underlying patterns to fill in the missing data. An enrichment stage of the original data, using spatial interpolation in the first stage, is needed for data completion to offer any advantages.
• The two-stage imputation strategy combining spatial interpolation (stage 1) with data completion (stage 2) for the misclassified instances from the first stage offers additional robustness in the predictions and has the potential to further improve accuracy compared to the approach implemented in stage 1. The degree of improvement depends on the quality of information available for the second stage, which is a function of both the size of the database as well as the degree of enrichment established in the first stage for the originally predominantly dry nodes.

It is important to recognize that a limitation of the present study is that the quantification of performance improvement focused solely on the predictive accuracy of the established surrogate models (calibrated using the imputed databases). However, as coastal hazard is ultimately represented and communicated through risk descriptions derived from these predictions, such as storm surge thresholds with prescribed annual exceedance probabilities, future work should examine the impact of the different imputation strategies on the accuracy of such downstream hazard estimates.