3.1. Time Series Comparison
The three numerical model results are first compared qualitatively in the time domain to each other and to EXP. For the sake of brevity, not all measured locations, but a selection of sensor locations is presented here. Sensor locations were selected to be representative for different areas along the flume with clearly different physical behaviours of the waves. In
Figure 4, the time series of
η are compared at measurement location WG04, representing the offshore waves between the wave paddle and foreshore toe; WG07, representing the wave shoaling and incipient breaking area; WG13, representing the surf zone; WG14, representing the inner surf zone and toe of the dike location; and WLDM02, representing the bore interaction area on the promenade. For clarity, the
ηLW time series are shown separately in
Figure 5. The time series of
Ux are compared in
Figure 6 at the ECM location on the promenade. For the numerical models, actually the depth-averaged horizontal velocity
is shown instead, as it was shown to deliver a better correspondence to EXP than
Ux for OF [
3]. The same was found to be the case for DSPH (not shown), and SW1L only provides
, since it is a depth-averaged model. In
Figure 7, the time series of
Fx are compared to the LC measurements, and in
Figure 8 the time series of
p are compared at the PS locations selected at approximately equidistant positions along the array (i.e., 0.28 m from PS01 up to PS09 and 0.24 m up to PS13).
From these figures, it is immediately clear that all three numerical models provide results that are very close to EXP. Especially for
η, differences appear to be very small with more significant differences in the surf zone (
Figure 4c,d) and on top of the dike (
Figure 4e). Further differences are revealed when comparing the
η time series of the LW components only. OF does not correspond as well to EXP in the offshore zone (
Figure 5a) compared to the other two numerical models. In the surf zone, however, OF shows better correspondence to EXP together with SW1L, while DSPH starts to diverge more from EXP for
t greater than approximately 120 s (
Figure 5c,d).
Reproducing
Ux appears to be more challenging than
η for all numerical models. Most of the positive
Ux peak values (i.e., flow towards the vertical wall) are reproduced, while some of the return flow durations (i.e.,
t for
Ux < 0) are modelled longer by OF than SW1L, with DSPH being in between (
Figure 6). Unfortunately, return flow velocities were often not captured by the ECM measurements in EXP, mostly by too thin flow layers (no data).
For the
Fx (
Figure 7) and
p time series (
Figure 8), differences become more distinctive. DSPH shows (small) negative or sub-atmospheric
p peaks, not observed in EXP, that occur before some of the dynamic impact peaks and mostly for the lowest PSs (
Figure 8a). Both OF and DSPH appear to underestimate most peak values for both
Fx and
p, while phase differences with EXP are most apparent for DSPH and SW1L.
3.2. Spatial Distribution of Wave Characteristics
The evolution of the root mean square wave height
Hrms, the SW and LW components (i.e.,
Hrms,sw and
Hrms,lw), and the mean surface elevation
(wave setup) over the wave flume up to the toe of the dike are compared in
Figure 9. All models agree on the general evolution of the
Hrms curves along the flume. The wave height slightly decreases from the wave paddle up to the toe of the foreshore, more so in case of OF than DSPH (with DSPH closest to EXP): from the wave paddle location to the foreshore toe,
Hrms decreases about 10% more for OF than DSPH. SW1L shows a similar behaviour from its offshore boundary (i.e., at WG02) until the foreshore toe and corresponds most with DSPH. On the other hand, SW shoaling is overestimated by both DSPH and SW1L (
Hrms,sw at WG07 in
Figure 9). In the surf zone, DSPH reproduces the energy loss due to SW breaking best of all three numerical models (i.e., closest result to (R)EXP for
Hrms,sw at WG13 and WG14 in
Figure 9), while SW1L overestimates and OF underestimates
Hrms,sw there.
The LW wave height Hrms,lw evolution along the flume in the experiment is also reproduced by all three numerical models. An unexpected peak appears in the DSPH result near x = 126 m, which is not found in the results by OF and SW1L. Moreover DSPH significantly overestimates Hrms,lw at WG13, while OF underestimates Hrms,lw at the dike toe (WG14).
Overall, DSPH provides the best correspondence with (R)EXP for
Hrms followed by SW1L, while OF clearly underestimates it for all measured locations. In terms of the wave setup
, however, SW1L shows the best correspondence with (R)EXP, while OF and DSPH, respectively, over- and underestimate it until at least WG07. In the inner surf zone, OF corresponds better with (R)EXP for
together with SW1L, while DSPH significantly overestimates it (
at WG13–14 in
Figure 9).
3.3. Model Performance and Pattern Statistics
More than a qualitative validation and evaluation of inter-model performance is not possible with the time series comparison of
Section 3.1, especially when visually almost no discernible and consistent trends of distinction between the model results can be made. The model performance and pattern statistics, provided in
Appendix B and
Section 2.4, then become very useful for a quantitative evaluation. As
dr provides a single dimensionless measure of average error, it is suitable to provide insight into the spatial variation of model error in the flume. In addition to the
dr of each numerical model, the
dr of the repeated experiment (REXP) is also included in this analysis. A numerical model
dr higher than the
dr of REXP means that the numerical error cannot be reduced further compared to the experimental repeatability error and a near “perfect” model performance would be achieved with regard to the experiment [
3]. Therefore, a relative refined index of agreement
d’r (Equation (A8)) and a corresponding rating (
Table A1) was defined by Gruwez et al. [
3] which provides the performance of the numerical model relative to the experimental model uncertainty.
Table 2 and
Table 3 provide the
dr-values and the pattern statistics at key locations (dike toe and on the promenade, respectively). It is noted that the statistics for
η reported in
Table 3 were averaged over the four measured locations (WLDM01—WLDM04), for the sake of brevity and because it better represents the statistics for the processes on the promenade. The evolution of
dr and
R at the WG locations along the wave flume up to the toe of the dike is shown, respectively, in
Figure 10 and
Figure 11, for
ηSW (
dr,sw and
Rsw), for
ηLW (
dr,lw and
Rlw) and for
η (
dr,tot and
R), and of
dr for
η and
Ux on the dike in
Figure 12.
Offshore, DSPH has the best model performance (WG02-04 in
Figure 10, rated Excellent) followed by OF (rated Very Good), and this continues to be so up to the shoaling zone (WG07), although the rating for DSPH drops slightly to Very Good. On the other hand, while SW1L starts offshore with a (relative) model performance similar to OF (rated Very Good), a notable decrease in (relative) model performance occurs in the shoaling zone (WG07, rated Good). All models show a generally decreasing trend of
dr,tot over the surf zone (WG07-13) and increases back up to the dike toe (WG13-14). Over the surf zone, DSPH gradually becomes the least performing numerical model (WG13-14, rated Good) followed by SW1L (WG13, rated Good). The relative model performance of SW1L increases back to Very Good at the dike toe (WG14). The performance of OF is not as good as DSPH in the offshore area (WG02-04), but becomes the highest of all three numerical models in the surf zone (WG13-14, rated Very Good) and continues to perform the best on the dike as well (
Figure 12), where the
dr of
η remains more or less constant for all models, with exception of DSPH which increases slightly back to a rating of Very Good.
Separating
η into the SW and LW components reveals that
dr,sw mostly follows the same trend as
dr,tot, with the exception that DSPH performs better than SW1L at the dike toe for
dr,sw (
Figure 10). On the other hand,
dr,lw clearly has a different behaviour: OF does not reproduce the incident LWs as good as DSPH and SW1L, but its LW performance steadily increases towards the dike toe (
Figure 10), where the LW energy increases (
Figure 9). SW1L shows the overall best LW performance as it shows similar
dr,lw values offshore to DSPH and similar values to OF in the surf zone. It is also revealed that the increase in SW1L error at WG07 is mostly caused by a decrease in
ηsw performance. Even though SW1L shows the least performance in modelling
ηsw over the foreshore, that does not seem to affect its capability of reproducing the LW shoaling and energy transfer from the SW to LW components, with a similar accuracy to OF for modelling
ηlw in the surf zone. Increased accuracy of SWASH of the SW modelling can be obtained however, with increased vertical resolution: the SW8L model exhibits much better performance in the shoaling zone (i.e., SW1L:
dr,sw,WG07 = 0.73, SW8L:
dr,sw,WG07 = 0.86, not shown), and attains the same model performance for
η at the toe of the dike as DSPH (
dr,sw,WG14 = 0.53). However, because of the smaller wave height of the SW components at the dike toe compared to the LW components (
Figure 9), this improvement only slightly increases the overall model performance at the dike toe (
Table 2, SW1L:
dr,sw,WG14 = 0.85, SW8L:
dr,sw,WG14 = 0.86).
The pattern statistics
B* and
in
Table 2 represent, respectively, the accuracy of the wave setup and wave amplitude at the toe of the dike [
3], and spatial information of these errors could already be derived implicitly from the
and
Hrms graphs in
Figure 9. Both were already discussed in
Section 3.2. The result is that at the toe of the dike, DSPH has the best result of the three numerical models in terms of reproduction of the wave height (
Table 2,
= 1.02), but the worst result in terms of the wave setup (
Table 2,
B* = 0.21). OF has the worst result for the wave height (
Table 2,
= 0.89), while delivering a close second-best result with SW1L for the wave setup (
Table 2,
B* = 0.05). SW1L provides the lowest wave setup error (
Table 2,
B* = 0.04).
However, in the previous, spatial information about the accuracy of the wave phase modelling is missing and is shown separately in
Figure 11. From this figure it is clear that DSPH introduces the largest error in wave phases over the surf zone up to the dike toe, and that the error is mostly due to phase errors in the SWs. Additionally, an important contribution of phase error is present in the
ηLW result of DSPH as well, which is not observed in the other numerical model results. Consequently, at the toe of the dike the phase error is largest for DSPH (i.e., lowest
R value in
Table 2). However, at the dike toe the difference with SW1L is small, while OF provides the best phase correspondence with EXP.
On top of the dike, the
dr of
Ux is provided in
Figure 12 and
Table 3, and indicates a lower model performance for all three numerical models than obtained for
η. However, for the relative model performance
d’r this difference significantly reduces, so that the same rating is obtained for
Ux as for
η in case of DSPH and OF (
Table 3, rated Very Good). SW1L (and SW8L) has the lowest
d’r for
Ux and rating (
Table 3, rated Good). Although the wave setup at the dike toe is overestimated by each numerical model (
Table 2,
B* > 0),
η on the promenade is generally underestimated and
Ux as well (
Table 3,
B* < 0). The bore wave height is best represented by OF (indicated by
), closely followed by DSPH and SW1L. Phase differences are observed for all numerical models (i.e.,
R < 1.00 in
Table 3), but are lowest for OF, followed by DSPH and SW1L.
Next the
dr-values of the pressures at the vertical wall are compared in
Figure 13 and the statistics in
Table 4. Again, all models show a lower model performance than REXP, and OF obtained the highest value followed by SW1L and DSPH, both of which have very similar model performance along the PS array. Model performances of all models tend to decrease and converge to each other towards higher sensor locations on the vertical wall.
The
dr value of
Fx for each model is shown in an overview table (
Table 4) together with the other pressure and force related statistics. Even though OF has the highest model performance in terms of the
Fx time series (
Table 4,
dr,Fx = 0.76), the force peaks are better estimated by SW1L (
Table 4,
dr,Fx,max,OF = 0.85 and
dr,Fx,max,SW1L = 0.88), while it has the largest errors in the
Fx time series (
Table 4,
dr,Fx = 0.64). Moreover, SW1L underestimates the total impulse much more than OF does (
Table 4,
I*OF = 0.85 and
I*SW1L = 0.62). DSPH has a similar model performance as OF for the force peaks
Fx,max, while
I* is in between OF and SW1L and its overall model performance for
Fx is similar to SW1L. Consequently, the relative model performance for
Fx is rated Very Good for OF and Good for DSPH and SW1L. While the model performance slightly increases for SW8L compared to SW1L at the dike toe (
Table 2) and on the promenade (
Table 3), this does not translate into a better model performance for the wave impact on the vertical wall; in fact, almost every
Fx statistic is lower for SW8L than for SW1L (
Table 4).
Pattern statistics (
B*,
and
R) are included as well in
Table 4. They indicate that all numerical models underestimate the wave impact force and exhibit phase differences. OF shows the least overall underestimation (i.e.,
B* closest to zero) and the least phase differences (i.e., highest
R), while DSPH has the highest
value. The results for
Fx are slightly worse for the multi-layer model SW8L compared to the single layer model SW1L, except for
.
3.4. Skill Target Diagrams
After the spatial inter-model comparisons of
Section 3.1,
Section 3.2 and
Section 3.3 based on the model performance and pattern statistics, the pattern statistics are visualised here together in a skill target diagram as described in
Section 2.4 (
Figure 14). The selection of observed locations that is considered for these diagrams, is the same as that for the time series plots (
Section 3.1). This is to prevent as much as possible a biased evaluation of the model skill in the target diagram, because a particular area had more sensors (i.e., the offshore area for
η and the lower half of the pressure sensor array for
p). One exception is
η on the promenade, for which the pattern statistics were averaged over the four measured locations (WLDM01–WLDM04), and therefore the values from
Table 3 are used here as well. The general model performance is visualised by a circle with a radius equal to the mean of the ST skill scores (Equation (3)) or distances from the origin of each model data point in the target diagram of either
η (
Figure 14a) or
p (
Figure 14b). The repeated experiment (REXP) is visualised in the skill target diagrams as well, but only by the mean skill circle for
η and
p. This circle is included to have a reference of the experimental repeatability error. For both
Ux and
Fx only one representative observed location is available. They are visualised as pentagrams together with the model data points for
η and
p, respectively.
None of the mean skill circles of the three numerical models have a smaller radius than the REXP circle, which means that none of the models has an ideal model skill relative to the experimental repeatability. In case of η, the model skill circle of DSPH is largest, and therefore DSPH has the lowest overall model skill, followed by the smaller circles of SW1L and OF (highest model skill), respectively. However, the Ux pentagrams suggest a better Ux model skill of DSPH than SW1L. For p, OF remains the numerical model with the highest model skill, followed by DSPH and SW1L. The Fx pentagrams have the same ranking.
In the
η skill target diagram, the numerical model skill for the location on the dike is indicated by an arrow. The remaining numerical model skill scores are those of the measured locations along the flume up to the dike toe. For OF, they are positioned in the top left quadrant, which means that the wave energy is underestimated (
< 0) and the wave setup is slightly overestimated (
B* > 0) (both confirmed by
Figure 9). For DSPH, the wave energy is mostly overestimated (
> 0). The two points furthest removed from the origin are the measured locations in the surf zone (i.e., WG13-14), where high
B* and low
R-values (or wave phase differences) are the largest contributors to the decreased DSPH model skill in this area. SW1L generally shows an overestimation of the wave energy (
> 0) and increased wave phase differences (lower
R) in the shoaling and surf zones. Generally, SW1L has the best wave setup results (lowest
B* values). The
Ux skill of all numerical models is one of a clear underestimation (pentagrams in lower left quadrant), of both the mean value (B* < 0) and standard deviation (
< 0), and increased phase difference (low
R-values). The same is valid for
p and
Fx, where all numerical model skill points are also positioned in the lower left quadrant.
3.5. Snapshot Inter-Model Comparison
The numerical models applied in this paper typically have a higher spatial resolution of the physical parameters of interest (e.g.,
η,
U,
p) than an experiment. This allows a comparison of snapshots between the numerical models. To allow a comparison of the velocity field, the multi-layered model SW8L is used instead of SW1L. The first main impact series is most appropriate for such a comparison, because accumulation of errors is lowest at the beginning of the test. Snapshots of the flow on the dike and the pressure distribution along the vertical wall are compared in
Figure 15,
Figure 16 and
Figure 17. A few key time instants in the
Fx and
Ux time series were selected during this series of impacts and are listed chronologically in
Table 5. These time instants were selected from each model result independent of time, because due to phase errors these key time instants have occurred at (slightly) different times between the models.
The first main impact was identified by Gruwez et al. [
3] to be caused by a plunging breaking bore pattern impacting on the vertical wall. The overturning wave arose when a large incoming bore collided with a smaller bore that was reflected against the vertical wall only a few moments before. This collision occurred at different locations on the promenade for each model and explains the timing mismatch of the
Fx impact peak with EXP (see
Fx graph insets in
Figure 15c). The timing of the smaller bore impact corresponds with EXP in case of OF and SW8L, but is late for DSPH (time instant 1,
Figure 15a). For all numerical models, the large incoming bore arrived later than was observed in EXP. This means that the collision between the reflected small bore and incoming large bore was timed differently for each model, with repercussions for the subsequent impact of the overturned wave on the vertical wall.
The best result is obtained by the OF model, which modelled a correct timing of the smaller bore reflection against the wall, but the late arrival of the larger incoming bore (i.e., by approximately 0.3 s) caused the collision to occur further from the wall than in the EXP (time instant 2,
Figure 17a). Nevertheless, for the impact on the wall (time instants 3 and 4,
Figure 15b,c), OF is able to—albeit mostly qualitatively—reproduce the shape of the pressure distribution which is distinctly different from a hydrostatic pressure distribution: both the pressure peak at PS10 for time instant 3 and the general shape of the pressure distribution at time instant 4 are captured by the model. We direct interested readers to the work in [
3] for a more detailed description of this comparison. DSPH has a very similar result, but the timing of the smaller bore was late as well (i.e., by approximately 0.7 s), meaning that the collision with the larger bore occurred closer to the wall than observed in EXP (time instant 3,
Figure 15b). Although the model did not capture the pressure peak at PS10 for time instant 3, it did manage to approach the pressure distribution qualitatively as well during the dynamic impact (time instant 4,
Figure 15c). Although SW8L managed to get the timing of the smaller bore impact right (time instant 1,
Figure 15a), the larger bore impacted the wall much later than in EXP (i.e., by approximately 1.2 s). This means that no interaction between the bores was modelled on the promenade. In any case, SWASH is a depth-integrated model, so it is inherently not able to model overturning waves explicitly. Moreover, only hydrostatic pressure distributions are provided by the model to avoid spurious numerical oscillations (
Section 2.3). However, even with those limitations, SW8L (and SW1L) still managed to predict an
Fx peak during the dynamic impact (time instant 4,
Figure 15c), but the pressure distribution remains hydrostatic and therefore no local pressure peaks were captured at all.
After the dynamic impact (time instant 4), the bore ran up the vertical wall (time instant 5,
Figure 15d) and reflected, causing a second quasi-static
Fx peak (time instant 6,
Figure 16a). In both the OF and DSPH results, a clockwise vortex formed near the bottom of the wall during the run-up process. However, in case of DSPH, this vortex was much stronger and lasted during the quasi-static
Fx peak as well. The
p distribution of EXP was mostly hydrostatic, except for a small local peak at PS06 (time instant 5), which seems to have been captured qualitatively by DSPH. On the other hand, the strong vortex modelled by DSPH also caused a non-hydrostatic
p distribution during time instant 6, while it was mostly hydrostatic in EXP. In this time instant, again OF was closest to the EXP observation. SW8L was not successful in correctly predicting the wave run-up against the vertical wall during reflection of the large impacting bore (time instant 5) and consequently underestimated
p and
Fx more than the other numerical models for time instants 5 and 6. During return flow (time instant 8,
Figure 16b) the pressure distribution was mostly hydrostatic, and all numerical models were able to predict the
p distribution well. Only DSPH shows a pressure decrease near the bottom of the wall (PS01), possibly caused by the persistent vortex modelled there, but now further removed from the wall compared to the previous time instants 5 and 6.
In terms of
U, generally similar velocity field patterns are found for all three numerical models with differences mostly explained by (small) phase differences of the individual bores interacting on the promenade or limitations in the physics it can represent (in case of SWASH). Considering the velocity profile at the ECM location during a maximum incident
Ux event (time instant 2,
Figure 15b) followed by a maximum return flow
Ux event (time instant 7,
Figure 17b), it can be seen that all numerical models underestimate
Ux of EXP close to the bottom for the incident bore (time instant 2), while OF predicts it very well, DSPH slightly underestimates it, and SW8L overestimates it during the return flow (time instant 7).
3.6. Computational Cost
Not only the model performance is of importance in practical applications of numerical models, but also the computational cost that it requires. An overview is provided in
Table 6 of the model resolution, total amount of grid cells/particles and corresponding computational cost for the computational hardware applied. The numerical convergence analyses in
Appendix A showed that the main characteristics of
η at the toe of the dike would not change more than 5% by increasing the grid or initial particle distance resolution beyond the values provided in
Table 6. For OF, this was achieved by Δ
x =
Δz =
H/20 =
Lm,t/260 (with
H the wave height and
Lm,t the mean wave length of the SW components at the dike toe), for DSPH by
dp =
H/50 =
Lm,t/585 and for SWASH by Δ
x =
Lm,o/170 =
Lm,t/60 (with
Lm,o the mean wave length of the SW components offshore).
Because of its Lagrangian description of the NS equations, DSPH has the advantage that it can be highly parallelised and is therefore able to make use of the many computational cores available in GPUs. On the other hand, OF and SWASH can be run in a parallelised way as well, but only on CPU cores, which are typically much less numerous. Different hardware and different amounts of cores are applied for each model, so only a qualitative comparison of the computational cost is possible. However, the applied hardware is in each case currently representative of what is typically available at research labs. OF and DSPH were run on multiple cores (CPU and GPU respectively) and SWASH on a single core.
Because the flow in the vertical dimension is fully resolved by OF and DSPH in addition to the horizontal dimension and at a much higher resolution, their computational cost is significantly higher than for the depth-averaged SW1L model. The increase of calculation time compared to SW1L is about 5000 times in case of OF and 1000 times for DSPH even though DSPH has more than 4 times the computational points than the OF model grid. Adding eight vertical layers to the SWASH model (SW8L) leads to a factor 7 increase in calculation time (on a single core machine). Although it will not affect this conclusion, it should be noted, however, that the SWASH computational domain is about 25% shorter compared to OF and DSPH, because waves are generated at the WG02 location in the SWASH model (
Figure 3).
In the computational cost, the model setup time should also be included. This depends on the experience of the practitioner, so the model setup time is only discussed here in general terms based on experienced practitioners for each respective model. SWASH is the most straightforward in this respect and requires the least “hands-on” time. The model setup of DSPH is also quite straightforward, where SPH particles are initially created on nodes of a regular lattice in a few seconds. The model domain boundaries and the water volume are defined by particles in DSPH, while an intricate mesh is needed for Eulerian models such as OF. Fortunately, the mesh generation for OF has been made much smoother thanks to automatic mesh generation algorithms such as cfMesh as applied here. Still, OF is found to require the most model setup time (i.e., typically more than double). This is certainly the case when a variable grid resolution is used because the refinement zones need to be defined beforehand, and this can only be done accurately by having a reliable estimation of the location of the surface elevation. This is most easily obtained from a fast SWASH model result (at least for the wave propagation until the dike toe), or alternatively by introducing refinement zones iteratively between OF model runs.