# Uncertainty Estimation in Hydrogeological Forecasting with Neural Networks: Impact of Spatial Distribution of Rainfalls and Random Initialization of the Model

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Material and Methods

#### 2.1. Neural Network Models

#### 2.1.1. Definitions

#### 2.1.2. Role of Time in Neural Networks Models

_{r}the sliding time-window size that defines the length of the necessary history of exogenous data; and W the matrix of parameters.

#### 2.1.3. Training and Overfitting

#### 2.1.4. Regularization Methods

_{X}represents the median calculated on the outputs of a set of X models. The choice of the number of models in the ensemble depends on the application.

#### 2.1.5. Model Design

- -
- The window widths of the different (exogenous) input variables (n
_{r}in Equations (1)–(3)). - -
- The “order” of the model, corresponding to the window width of the estimated (or observed, if the model is a feed-forward) targeted variable (output variable), for previous time-steps, applied at the input of the model (r in Equations (2) and (3)).
- -
- The number of neurons in the hidden layers: N.

#### 2.2. Study Area: The Champagne Chalk Groundwater Basin

#### 2.2.1. Field Study Presentation

- LocationLocated in Northern France, in the Grand-Est region, the Champagne chalk groundwater basin area is estimated at 5927 sq.km. It corresponds mainly to the drainage of the rivers Marne and Aube, delimited by piezometric ridges characterized as follows: chalk limit on the eastern part, tertiary rocks on the western part, other hydrogeological basins on the northern limit, the Seine river for the southern part and, as a bedrock, marlstones [23]. Elevation varies from 40 to 286 m.a.s.l. (Figure 2).
- Water useWater is mainly used for tap water production and agriculture [23]. Annual water withdrawals via studied piezometer made on average between 2012 and 2017 are 17,393 m
^{3}, however, showing a decreasing trend [24]. Water is also used for agriculture, with 61.5% of groundwater withdrawal for irrigation in 2017 (against 38.5% for tap water production) in Vailly (location of the studied piezometer) and neighboring towns [25]. - Climate
- Geology and groundwater behaviorThis basin is mainly composed of chalk, and limestones to a lesser proportion, with sands and clay along the hydrographic network [27,28]. Intense shallow fracturing, mainly caused by climate action, has developed a significant permeability especially near the hydrographic network. Groundwater recharge time in the champagne basin is estimated at 100 days in our study piezometer (Craie à Vailly (nouveau)) [29], and the underground levels can increase from 6 m to 25 m [23,30]. Groundwater levels, especially in the Barbuise catchment area, which is close to the study piezometer, are influenced by the shallow water [27]. Consequently, the Barbuise river discharge is strongly correlated to piezometric levels at Craie à Vailly [27,29].

#### 2.2.2. Database Presentation

- Troyes-Barberey (R
_{TB}) (precipitation and potential evapotranspiration), - Grandes-Chapelles (R
_{GC}) (precipitation), - Mailly (R
_{MA}) (precipitation)

_{CV}), and two discharge stations are located in Pouan-les-Vallées in Barbuise catchment (D

_{BP}) and at Méry-sur-Seine in the Seine catchment (D

_{SM}) (Table 1).

_{CV}) piezometer, it appears that the shortest response time is two time-steps for the discharge at Barbuise at Pouan les Vallées (D

_{BP}). This indicates that, statistically, the discharges at Barbuise at Pouan les Vallées have a greater influence on the groundwater at Craie at Vailly after two time-steps delay. And that this response time is the shorter. This confirms the quick interaction between surface water and groundwater. Regarding the impact of surface water on both the water quality and the groundwater level, the two time-step lead-time was thus chosen. In this way, a lead-time of 20 days (two time-step) is considered as a good compromise between model accuracy and end users’ needs. A shorter lead-time would reduce the interest of the forecast for the end users, while a longer lead-time would require the availability of the Barbuise at Pouan les Vallées discharge forecast. Thus, this lead time ensures that available inputs explain the output.

#### 2.3. Quality Criteria

#### 2.3.1. Quality of Fitting and Prediction

- The persistency criterion

#### 2.3.2. Uncertainties Quantification

- Prediction Interval Coverage Probability

- Mean Prediction Interval

_{MPI}, is the average of all the results set of the interval of prediction calculated at each time-step. It quantifies the mean scattering of the prediction [34], following (8).

- Prediction Confidence Criterion

_{PC}, is a ratio quantifying the performance of a predictor for providing a prediction having the highest empirical probability of lying within the smaller prediction interval (9). It is simply defined by the ratio between the two previous criteria [13].

#### 2.4. Uncertainties Linked to the Initialization of the Parameters and to the Spatial Variability of the Rains

#### 2.4.1. Variability Due to the Initialization of Parameters

_{PC}criterion (Prediction Confidence Criterion) [13] defined in Section 2.3.2, that synthetizes both criteria.

#### 2.4.2. Spatial Rainfall Variability

_{-RG}. All the permutations made give a range of outputs that can be considered as a prediction interval related to the spatial variability of rain.

#### 2.5. Estimation of Empirical Confidence Intervals Using Probability Density Functions

#### 2.5.1. Method

- -
- Establishing the frequencies of appearance of the water level classes histogram; this is then considered as an empirical probability density function (pdf) of the data;
- -
- Fitting a theoretical well-known pdf, for example the normal one, to the empirical pdf by adjusting its parameters. If necessary, thanks to the Expectation-Maximization algorithm (EM) [36,37], the theoretical pdf can be a composition of several pdfs of the same type, each one having different parameters; this composite pdf is called the target pdf. The algorithm provides the constituent parameters of the theoretical elementary theoretical pdfs as well as the weights that enable them to be assembled to fit the target pdf;
- -
- Starting from target pdf, determining a probability of occurrence of the measured value inside the predicted interval for each class;
- -
- For a given confidence index (for example 95%), and for each class, supposing the data verify the constraints of a normal law and establishing a model of “correctness” using the erf (error function). This provides the estimated error associated to each class;
- -
- Finally, drawing the possible errors on the water chart.

#### 2.5.2. Chosen Probability Density Functions

## 3. Model Design

#### 3.1. Definition of Subsets for Training Testing, Stop and Cross-Validation

_{P}the score of persistency, and q the number of the considered subset.

#### 3.2. Choice of the Model and Complexity Selection

## 4. Results

#### 4.1. Optimal Number of Members in Ensemble Models

_{PC}, Prediction Confidence Criterion, is calculated for each ensemble, allowing definition of the optimal number of members, i.e., the number of members whose parameters are randomly initialized (X in Equation (4)).

_{PC}versus the number of members in the ensemble models. Schematically, the curve can be approximated by two straight lines whose intersection is at around 40 members. The first line decreases when the number of members in the ensemble increases, corresponding to a stage where the MPI increases. The second line corresponds to a plateau that indicates the stability of the two criteria that make up the C

_{PC.}The intersection of two lines corresponds to the minimal number of initializations for which the gain of ensemble starts to become stationary. We thus propose this value (40 members in Figure 5) as the number X of members. Although the C

_{PC}could possibly be enhanced by using more members, the cost-benefit ratio (especially regarding calculation time) pleads in favor of this choice.

#### 4.2. Prediction Results

_{p}= 0.65 and the test score is C

_{p}= 0.40. The performances are thus lower on the test set than on the other sets during cross validation. This is consistent with the choice of the test set, which corresponds to the drier period of the database. Nevertheless, the quality of the forecast presented in Figure 6, made for the year with the driest summer in the entire database, shows that: (1) the model is capable of generalizing to periods of extreme behavior, (2) confirms the interest in being able to visualize uncertainty, so that the manager can analyze the most uncertain parts of the limnigram. As a reminder, the three last months of 1990 were not considered due to possible errors in piezometric levels.

_{ICP}= 0.39, M

_{PI}= 0.62 m, and C

_{PC}= 0.63 m

^{−1}. It appears that the prediction is fairly close to the measurement except for the early spring of 1990, for which the forecast level is not low enough. On the other hand, the grey band showing the uncertainty is very thin and does not contain enough observed values (P

_{ICP}= 0.39) to be able to inspire confidence in the end users.

#### 4.3. Representation of Uncertainties Caused by the Initialization Parameters

#### 4.3.1. Theoretical Composite pdf for Four Distributions

_{PICP}on the Train+Stop dataset. To this end Figure 8 shows the representation, explained in Figure 7, for each one of the theoretical laws, regarding the two distributions of values inside the prediction interval (green) or outside this interval (red). Table 6 shows the correlations between the empirical distribution and the theoretical laws, for the measured groundwater levels inside the prediction interval $\left({r}_{in}^{2}\right)$ and outside the prediction interval $({r}_{out}^{2})$. Best correlations are shown in green and worst in red. It appears in Table 6 that the best adjustment is achieved by the Raised Cosine theoretical law.

_{PICP}= 0.51, meaning that the probability of the interval of prediction containing the observed value is similar to the probability of it not containing the observed value, whatever the groundwater level and theoretical composite pdf law selected. We can also notice that Normal, Bhattacharjee and Huber laws have the same kind of pattern whereas Logistic, Gumbel and Raised Cosine laws have similar shapes. This can be explained by the fact that Slash, Bhattacharjee and Huber are derived from the Normal law. Logistic, Gumbel and Raised Cosine laws seem to fit well with observed groundwater level distribution inside the prediction interval, having a Pearson’s correlation coefficient over 0.74 for measured water levels inside the prediction interval, whereas other laws provide correlations ranging from 0.50 to 0.69 (Table 6 and Figure 8).

_{PICP}= 0.24, meaning that the observed groundwater levels outside the prediction interval are more numerous than groundwater levels inside the prediction interval. Pearson’s correlation coefficients between the distribution of groundwater levels and the composite theoretical laws are shown in Table 7.

^{2}varying between 0.52 and 0.57. Slash, Raised Cosine and Logistic laws seem to provide the best correlated composite pdf, with a correlation between 0.61 and 0.66. On the other hand, the composite pdf representing observed values inside the prediction interval has two flared “peaks” at 117 m.a.s.l. and 127 m.a.s.l. for the nine laws. However, correlations are low due to the small frequencies of increasing groundwater levels inside prediction interval. Laplace and Cauchy laws appear to be the laws with the best fit, with a correlation above 0.45. Raised Cosine and Logistic correlations reach only 0.44.

#### 4.3.2. Error Margins

- -
- It is supposed that the distribution of samples inside a class follows a Normal Distribution,
- -
- When a class contains no sample, for example, the class around 135 m.a.s.l., the error is maximum and is divided into two parts: 50% above 50% underneath the probability.
- -
- When a class contains very few samples (less than three), this class is not considered for r
_{C}^{2}and M_{E}calculations.

_{PICP}(each cross is a C

_{PICP}calculated thanks to the ensemble model), always having a Pearson’s correlation under 0.3. This is consistent with the high dispersion of C

_{PICP}. However, the percentage of these C

_{PICP}included inside the calculated error margin seems to be a better indicator of the quality of the model of correctness. In this case, six laws have more than 75% of C

_{PICP}inside the error margin. Pearson’s correlation coefficients and the error margin indicator for Cauchy and Slash law’s models of correctness are the highest, with, respectively, 0.25 and 75% and 0.22 and 80.3% values (Table 8).

_{PICP}and the model of correctness are still low, with an average value around 0.2 for all laws. The highest correlation comes from the model of correctness provided by the Slash mixed pdf. However, the crosses representing the C

_{PICP}inside the error margins, reaching more than 79% for Logistic law (Table 9), are slightly higher than the ones obtained for decreasing groundwater levels (Table 8). Figure 11 shows that the models of correctness of the nine laws have a similar shape, with a stagnation of probability for observed piezometric levels above 125 m.a.s.l. The probability of correctness, for each law and each groundwater level, is above or equal to 0.5.

#### 4.4. Determination of Spatial Distribution of Rainfall Uncertainty

_{PICP}equals 0.49 for the test set, shown in Figure 12. This was 0.39, considering only variability due to the parameter’s initialization. Regarding the prediction interval, it logically became wider when including the rainfall variability, with C

_{MPI}criterion of 0.73 m and a C

_{PC}of 0.68 m

^{−1}.

#### 4.5. Impact of the Spatial Distribution of Rainfall Uncertainty on the Model of Correctness

#### 4.6. Definition of a Confidence Interval

_{PICP}are gathered in Table 11, and Figure 13 shows the confidence interval obtained for the confidence index of 0.90.

_{MPI}decreases more quickly than the confidence index, which allows the manager to choose a compromise according to his requirements.

## 5. Discussion

#### 5.1. Role of Rain in the Forecast Interval

_{MPI}= 0.62 m (2.5%) for the first and, respectively, C

_{MPI}= 0.73 m (3%) for the second. These small intervals seem to be very accurate, but they do not provide any real added value for the user, since the measured water level does not always fall within this interval. On average, the C

_{PICP}provides the probability that the prediction interval from the model contains the measured value; this is 39% for the former and 49% for the latter. Thus, even considering the uncertainty caused by the measurement of rainfall variability (Figure 12), the model is only correct, on average, one time out of two.

_{BP}) is 2 decades; however, in Figure 6, we can see that the most important rainfall episode of February 1990 (p > 60 mm) influences the water table in less than one decade, its maximum effect appearing at 2 decades. It thus appears that infiltration with faster dynamics occurs during heavy rainfall episodes and that the model has difficulties in representing these fast and rare infiltrations. Moreover, the prediction interval is rather smaller for the responses to these episodes than for the other configurations, both for Figure 6 and Figure 12, suggesting that, during very wet episodes, the spatial variability of the rainfall events, at the decadal step, does not have a great impact on the response. Given the objective of the modelling, which is to predict low water, this double property, errors in prediction and low uncertainty during high rain pulses, can be considered as not being prohibitive.

#### 5.2. Role of the Amount of Data

## 6. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## Appendix A

**Figure A1.**Details of the construction and of the values of the irrigation inputs for the different types of crops as a function of the surface, the water requirement and the season (by months).

## References

- Hartmann, D.L.; Klein Tank, A.M.G.; Rusticucci, M.; Alexander, L.V.; Brönnimann, S.; Charabi, Y.A.-R.; Dentener, F.J.; Dlugokencky, E.J.; Easterling, D.R.; Kaplan, A.; et al. Observations: Atmosphere and Surface. In Climate Change 2013: The Physical Science Basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change; Cambridge University Press: Cambridge, UK, 2013; Chapter 2; pp. 159–254. [Google Scholar]
- Hornik, K.; Stinchombe, M.; White, H. Multilayer Feedforward Networks are Universal Approximators. Neural Netw.
**1989**, 2, 359–366. [Google Scholar] [CrossRef] - Wagener, T.; Montanari, A. Convergence of approaches toward reducing uncertainty in predictions in ungauged basins. Water Resour. Res.
**2011**, 47. [Google Scholar] [CrossRef][Green Version] - Bourgin, F. Comment Quantifier l’incertitude Prédictive en Modélisation Hydrologique? Travail Exploratoire sur un Grand Échantillon de Bassins Versants; AgroParisTech: Paris, France, 2014. [Google Scholar]
- Solomatine, D.P.; Shrestha, D.L. A novel method to estimate model uncertainty using machine learning techniques. Water Resour. Res.
**2009**, 45, 1–16. [Google Scholar] [CrossRef] - Biondi, D.; Todini, E. Comparing Hydrological Postprocessors Including Ensemble Predictions into Full Predictive Probability Distribution of Streamflow. Water Resour. Res.
**2018**, 54, 9860–9882. [Google Scholar] [CrossRef][Green Version] - Krzysztofowicz, R. Bayesian theory of probabilistic forecasting via deterministic hydrologic model. Water Resour. Res.
**1999**, 35, 2739–2750. [Google Scholar] [CrossRef][Green Version] - Biondi, D.; Luca, D.L.D. Performance assessment of a Bayesian Forecasting System (BFS) for real-time flood forecasting. J. Hydrol.
**2013**, 479, 51–63. [Google Scholar] [CrossRef] - Wu, W.; Dandy, G.C.; Maier, H.R. Protocol for developing ANN models and its application to the assessment of the quality of the ANN model development process in drinking water quality modelling. Environ. Model. Softw.
**2014**, 54, 108–127. [Google Scholar] [CrossRef] - Kong-A-Siou, L.; Johannet, A.; Borrell, V.; Pistre, S. Complexity selection of a neural network model for karst flood forecasting: The case of the Lez Basin (southern France). J. Hydrol.
**2011**, 403, 367–380. [Google Scholar] [CrossRef][Green Version] - Stone, M. Cross-Validatory Choice and Assessment of Statistical Predictions (With Discussion). J. R. Stat. Soc. Ser. B
**1974**, 38, 111–147. [Google Scholar] [CrossRef] - Sjöberg, J.; Zhang, Q.; Ljung, L.; Benveniste, A.; Delyon, B.; Glorennec, P.-Y.; Hjalmarsson, H.; Juditskys, A. Nonlinear Black-box Modeling in System Identification: A Unified Overview. Automatica
**1995**, 31, 1691–1724. [Google Scholar] [CrossRef][Green Version] - Akil, N.; Artigue, G.; Savary, M.; Johannet, A.; Vinches, M. Quantification of Neural Network Uncertainties on the Hydrogeological Predictions by Probability Density Functions. In Proceedings of the IOP Conference Series: Earth and Environmental Science, Prague, Czech Republic, 9–13 September 2019; p. 10. [Google Scholar]
- Dreyfus, G. Neural Networks, Methodology and Applications; Springer: Berlin, Germany, 2005; p. 509. [Google Scholar]
- Hornik, K. Approximation capabilities of multilayer feedforward networks. Neural Netw.
**1991**, 4, 251–257. [Google Scholar] [CrossRef] - Barron, A.R. Approximation bounds for superpositions of a sigmoidal function. In Proceedings of the IEEE International Symposium on Information Theory—Proceedings, San Antonio, TX, USA, 17–22 January 1993; pp. 930–945. [Google Scholar]
- Nerrand, O.; Roussel-Ragot, P.; Personnaz, L.; Dreyfus, G.; Marcos, S. Neural Networks and Nonlinear Adaptive Filtering: Unifying Concepts and New Algorithms. Neural Comput.
**1993**, 5, 165–199. [Google Scholar] [CrossRef] - Artigue, G.; Johannet, A.; Borrell, V.; Pistre, S. Flash flood forecasting in poorly gauged basins using neural networks: Case study of the Gardon de Mialet basin (southern France). Nat. Hazards Earth Syst. Sci.
**2012**, 12, 3307–3324. [Google Scholar] [CrossRef][Green Version] - Taver, V.; Johannet, A.; Borrell-Estupina, V.; Pistre, S. Feed-forward vs recurrent neural network models for non-stationarity modelling using data assimilation and adaptivity. Hydrol. Sci. J.
**2015**, 60, 1242–1265. [Google Scholar] [CrossRef][Green Version] - Geman, S.; Bienenstock, E.; Doursat, R. Neural Networks and the Bias/Variance dilemma. Neural Comput.
**1992**, 4, 1–58. [Google Scholar] [CrossRef] - Darras, T.; Johannet, A.; Vayssade, B.; Long-a-Siou, L.; Pistre, S. Influence of the Initialization of Multilayer Perceptron for Flash Floods Forecasting: How Designing a Robust Model. In International Work-Conference on Time Series 2014; Springer: Granada, Spain, 2014; p. 13. [Google Scholar]
- Kong-A-Siou, L.; Johannet, A.; Borrell Estupina, V.; Pistre, S. Optimization of the generalization capability for rainfall-runoff modeling by neural networks: The case of the Lez aquifer (southern France). Environ. Earth Sci.
**2012**, 65, 2365–2375. [Google Scholar] [CrossRef] - AESN. Fiche de Caractérisation de la ME HG20—Masse d’eau souterraine HG208 “Craie de Champagne Sud et Centre; AESN: Nanterre, France, 2015. [Google Scholar]
- Météo France. Données Publiques—Observations In Situ. Available online: https://donneespubliques.meteofrance.fr/?fond=rubrique&id_rubrique=26 (accessed on 8 September 2020).
- BNPE. Base de données des prélèvements en eau, OPR0000035289. Available online: https://bnpe.eaufrance.fr/acces-donnees/codeOuvrage/OPR0000035289/annee/2014 (accessed on 15 January 2021).
- BNPE. Base de Données des Prélèvements en eau, Vailly (10). Available online: https://bnpe.eaufrance.fr/?q=acces-donnees/codeCommune/10391/annee/2017/etCommunesAdjacentes (accessed on 31 May 2021).
- Stollsteiner, P. Connaissance des Ressources Réellement Disponibles sur L'ensemble des Bassins Versants Crayeux; BRGM: Reims, France, 2013. [Google Scholar]
- Kloppmann, W.; Dever, L.; Edmunds, W.M. Residence time of Chalk groundwaters in the Paris Basin and the North German Basin: A geochemical approach. Appl. Geochem.
**1998**, 13, 593–606. [Google Scholar] [CrossRef][Green Version] - Pinault, J.-L.; Allier, D.; Chabart, M. Prévision des Volumes d’eau Exploitables de 10 Bassins versants en Champagne Crayeuse; BRGM: Reims, France, 2006. [Google Scholar]
- Putot, E.; Verjus, P.; Vernoux, J.F. Qualification des Piézomètres du Réseau de Bassin Seine Normandie en 2005; BRGM: Massy, France, 2006. [Google Scholar]
- IGN. Registre Parcellaire Graphique (RPG): Contours Des Parcelles et Îlots Culturaux et Leur Groupe de Cultures Majoritaire. Available online: https://www.data.gouv.fr/fr/datasets/registre-parcellaire-graphique-rpg-contours-des-parcelles-et-ilots-culturaux-et-leur-groupe-de-cultures-majoritaire/#_ (accessed on 14 September 2020).
- Mangin, A. Pour une meilleure connaissance des systèmes hydrologiques à partir des analyses corrélatoire et spectrale. J. Hydrol.
**1984**, 67, 25–43. [Google Scholar] [CrossRef] - Kitanidis, P.K.; Bras, R.L. Real-time forecasting with a conceptual hydrologic model: 2. Applications and results. Water Resour. Res.
**1980**, 16, 1034–1044. [Google Scholar] [CrossRef] - Shrestha, D.L.; Solomatine, D.P. Machine learning approaches for estimation of prediction interval for the model output. Neural Netw.
**2006**, 19, 225–235. [Google Scholar] [CrossRef] - Khosravi, A.; Nahavandi, S.; Creighton, D. A prediction interval-based approach to determine optimal structures of neural network metamodels. Expert Syst. Appl.
**2010**, 37, 2377–2387. [Google Scholar] [CrossRef] - Dempster, A.P.; Laird, N.M.; Rubin, D.B. Maximum Likelihood from Incomplete Data via the EM Algorithm. J. R. Stat. Soc. Ser. B
**1977**, 39, 1–22. [Google Scholar] - Benaglia, T.; Chauveau, D.; Hunter, D.R.; Young, D.S. Mixtools: An R Package for Analyzing Finite Mixture Models. J. Stat. Softw.
**2009**, 32, 29. [Google Scholar] [CrossRef][Green Version] - De Moivre, A. The Doctrine of Chances, or, A Method of Calculating the Probability of Events in Play, III ed.; Chelsea Publishing Company: Londres, UK, 1756; p. 368. [Google Scholar]
- Fisher, R.A. On the mathematical foundations of theoretical statistics. Philos. Trans. R. Soc.
**1922**, 9, 309–368. [Google Scholar] - Gumbel, E.J. Méthodes graphiques pour l'analyse de débits de crues. Rev. De Stat. Appl.
**1957**, 5, 77–89. [Google Scholar] [CrossRef][Green Version] - Laplace, P.-S. Mémoire sur la probabilité des causes par les évènements. In Œuvres complètes de Laplace; Mémoires de l’Academie Royale des Sciences, Ed.; Divers Savan: Paris, France, 1774; Chapter 6; pp. 621–656. [Google Scholar]
- Rinne, H. Location-Scale Distributions Linear Estimation and Probability Plotting Using MATLAB; Justus Liebig University: Giessen, Allemagne, 2010. [Google Scholar]
- Sinha, R.K. A Thought on Exotic Statistical Distributions. Int. J. Math. Comput. Sci.
**2012**, 6, 49–52. [Google Scholar] - Cauchy, A.-L. CALCUL DES PROBABILITÉS—Sur les résultats moyens d’observations de même nature, et sur les résultats les plus probables. In Oeuvres Complètes; Cambridge University Press: Cambridge, UK, 1853; pp. 94–104. [Google Scholar]
- Stigler, S.M. An Historical Note on the Cauchy Distribution; Biometrika Trust: Oxford, UK, 1974; pp. 375–380. [Google Scholar]
- Verhulst, P.F. Recherches mathématiques sur la loi d'accroissement de la population. In Nouveaux mémoires de l'Académie Royale des Sciences et Belles-Lettres de Bruxelles; Nabu Press: Bruxelles, Belgique, 1845; Chapter 18; pp. 1–38. [Google Scholar]
- Rogers, W.H.; Tukey, J.W. Understanding some long-tailed symmetrical distributions. Stat. Neerl.
**1972**, 26, 211–226. [Google Scholar] [CrossRef] - Bhattacharjee, G.P.; Pandit, S.N.N.; Mohan, R. Dimensional Chains Involving Rectangular and Normal Error-Distributions. Technometrics
**1963**, 5, 404–406. [Google Scholar] [CrossRef] - Huber, P.J. Robust estimation of a location parameter. Ann. Math. Stat.
**1964**, 35, 73–101. [Google Scholar] [CrossRef] - Huber, P.J. Robust Statistics; Wiley: Hoboken, NJ, USA, 1981; p. 308. [Google Scholar]
- Levenberg, K. A method for the solution of certain non-linear problems in least squares. Q. Appl. Math.
**1944**, 2, 164–168. [Google Scholar] [CrossRef][Green Version] - Marquardt, D.W. An Algorithm for Least-Squares Estimation of Nonlinear Parameters. J. Soc. Ind. Appl. Math.
**1963**, 11, 431–441. [Google Scholar] [CrossRef]

**Figure 1.**Multilayer Perceptron representation, with x

_{i}, the exogenous variables;

**W**, the matrix of parameters; y, the measured output; ŷ, the predicted output; r, the order of the model, n

_{r}, the input window width; H

_{i}(i = 1 to N) the hidden neurons; N, the number of hidden neurons; k, the discrete time and h, the lead time [13].

**Figure 5.**C

_{PC}cross-validation score of ensemble models as a function of the number of members (X).

**Figure 6.**Prediction of groundwater levels at 20 days lead-time on the test set. The three last months of 1990 were not taken into account due to possible errors in piezometric levels. C

_{p}= 0.40; C

_{PICP}= 0.39; C

_{MPI}= 0.62 m.

**Figure 7.**Theoretical composite pdf of piezometric levels, obtained using Esperance Maximization (EM) algorithm using a Normal law and applied to observed values having a decreasing slope and being inside the prediction interval. Each elementary normal law is represented with a different color and denoted as ”EM component.”

**Figure 8.**Composite pdf for decreasing measured groundwater distributions with (

**a**) Normal law, (

**b**) Gumbel law, (

**c**) Laplace law, (

**d**) Raised Cosine law, (

**e**) Cauchy law, (

**f**) Logistic law, (

**g**) Slash law, (

**h**) Bhattacharjee law and (

**i**) Huber law.

**Figure 9.**Mixed pdf for increasing groundwater distributions. with (

**a**) Normal law, (

**b**) Gumbel law, (

**c**) Laplace law, (

**d**) Raised Cosine law, (

**e**) Cauchy law, (

**f**) Logistic law, (

**g**) Slash law, (

**h**) Bhattacharjee law and (

**i**) Huber law.

**Figure 10.**Models of correctness and Error margin (${M}_{E}$), calculated from composite pdf of increasing levels and using a confidence index of 95% for (

**a**) Normal law, (

**b**) Gumbel law, (

**c**) Laplace law, (

**d**) Raised Cosine law, (

**e**) Cauchy law, (

**f**) Logistic law, (

**g**) Slash law, (

**h**) Bhattacharjee law and (

**i**) Huber law.

**Figure 11.**Models of correctness and Error margin (${M}_{E}$), calculated from composite pdf of decreasing levels and using a confidence index of 95% for (

**a**) Normal law, (

**b**) Gumbel law, (

**c**) Laplace law, (

**d**) Raised Cosine law, (

**e**) Cauchy law, (

**f**) Logistic law, (

**g**) Slash law, (

**h**) Bhattacharjee law and (

**i**) Huber law.

**Figure 12.**Groundwater level forecasting with 20 days’ lead-time. Grey band shows the variability due to the parameter’s initialization and to the rain spatial variability. C

_{p}= 0.40; C

_{PICP}= 0.49; C

_{MPI}= 0.73 m.

**Figure 13.**Groundwater level forecasting for 20 days’ lead-time and confidence interval calculated with 0.90 confidence index. C

_{p}= 0.40; C

_{PICP}= 0.81; C

_{MPI}= 8.47 m.

Station Name | Measured Variable | Unit | Time Step | Max Value | Min Value | Median | Average |
---|---|---|---|---|---|---|---|

Craie at Vailly (L_{CV}) | Level | m.a.s.l. | 10 days | 134.75 | 109.75 | 119.95 | 120.558 |

Barbuise at Pouan les Vallées (D_{BP}) | Discharge | m^{3}.s^{−1} | 10 days | 4.50 | 0.00 | 0.67 | 0.836 |

Seine at Méry-sur-Seine (D_{SM}) | Discharge | m^{3}.s^{−1} | 10 days | 182.2 | 5.95 | 25.61 | 35.71 |

Grandes-Chapelles (R_{GC}) | Rain | mm | 10 days | 131.2 | 0.0 | 15.9 | 19.87 |

Troyes-Barberey (R_{TB}) | Rain | mm | 10 days | 86.4 | 0.0 | 13.2 | 17.29 |

Mailly (R_{MA}) | Rain | mm | 10 days | 138.8 | 0.0 | 17.0 | 21.50 |

Troyes-Barberey (PET) | Potential Evapo-transpiration | mm | 10 days | 64.7 | 0.0 | 19.0 | 21.20 |

Bassin (I) | Irrigation | m^{3}.ha^{−1}.month^{−1} | month | 833.9 | 0.6 | 176.0 | 280.1 |

**Table 2.**Correlation analysis. Diagonal shows the memory effect (in number of time-steps) when simple correlation is calculated (orange). When cross-correlation is calculated, blue cells show memory effect and green cells show response time. NC means that correlation score is always under 0.2, showing a very weak correlation, leading to possible misinterpretations of the memory effect.

L_{CV} | ΔL_{CV} | D_{BP} | D_{SM} | R_{GC} | R_{TB} | R_{MA} | PET | I | |
---|---|---|---|---|---|---|---|---|---|

L_{CV} | 17 | 6 | 2 | 5 | 21 | 22 | 15 | 12 | 16 |

ΔL_{CV} | 15 | 4 | −4 | 0 | 1 | 1 | 1 | 4 | 8 |

D_{BP} | 17 | 1 | 11 | 3 | 2 | 6 | 7 | 9 | 12 |

D_{SM} | 19 | 4 | 15 | 5 | 1 | 1 | 1 | 3 | 7 |

R_{GC} | NC | 2 | NC | 3 | 0 | 0 | 0 | 0 | 27 |

R_{TB} | NC | 2 | NC | 3 | 1 | 1 | 0 | 0 | 27 |

R_{MA} | NC | 3 | NC | 3 | 0 | 1 | 0 | 0 | 33 |

PET | 19 | 11 | 16 | 9 | NC | NC | NC | 8 | 4 |

I | 22 | 14 | 19 | 12 | NC | NC | NC | 11 | 7 |

Name of pdf Law | Formula | Eq. | References |
---|---|---|---|

Normal | $\mathcal{N}\left(\overline{x},\text{}{\sigma}^{2}\right)=\frac{1}{\sigma \sqrt{2\pi}}\text{}{e}^{-\frac{\left(x-\overline{x}\right)\xb2}{2\sigma \xb2}}$ | (10) | [38,39] |

Gumbel | $\mathcal{G}\U0001d4ca\U0001d4c2\left(\overline{x},\beta \right)=\frac{{e}^{-\frac{\left(x-\overline{x}\right)}{\beta}}{e}^{-{e}^{-\frac{\left(x-\overline{x}\right)}{\beta}}}}{\beta}$ | (11) | [40] |

Laplace | $\mathcal{L}\U0001d4b6\U0001d4c5\left(\overline{x},\text{}b\right)=\frac{1}{2b}{e}^{-\frac{\left(\left|x-\overline{x}\right|\right)}{b}}$ | (12) | [41] |

Raised Cosine | $\mathcal{C}\u2134\U0001d4c8\U0001d4c7\left(\overline{x},\text{}s\right)=\frac{1}{2s}\text{}\left(1+\mathrm{cos}\left(\frac{\left(x-\overline{x}\right)}{s}\text{}\pi \right)\right)$ | (13) | [42,43] |

Cauchy | $\mathcal{C}\U0001d4b6\U0001d4ca\left({x}_{0},\text{}a\right)=\frac{1}{\pi a\text{}\left(1+{\left(\frac{x-{x}_{0}\text{}}{a}\right)}^{2}\right)}$ | (14) | [44,45] |

Logistic | $\mathcal{L}og\U0001d4be\U0001d4c8\U0001d4c9\left(\overline{x},\text{}s\right)=\frac{{e}^{-\frac{\left(x-\overline{x}\right)}{s}}}{s{\left(1+{e}^{-\frac{\left(x-\overline{x}\right)}{s}}\right)}^{2}}$ | (15) | [46] |

Slash | $\mathcal{S}\ell \U0001d4b6\left(x\right)=\frac{\phi \left(0\right)-\phi \left(x\right)}{{x}^{2}}$ | (16) | [47] |

Bhattacharjee | $\mathcal{B}\U0001d4bd\U0001d4b6\U0001d4c9\left(\overline{x},{\sigma}_{x},\text{}a\right)=\frac{1}{2a}\left(\mathsf{\Phi}\left(\frac{x-\mathsf{\mu}+a}{\sigma}\right)-\mathsf{\Phi}\left(\frac{x-\mathsf{\mu}-a}{\sigma}\right)\right)$ | (17) | [48] |

Huber | $\mathscr{H}\U0001d4ca\U0001d4b7\left(z\right)=\frac{1}{2\sqrt{2\pi}\text{}\left(\mathsf{\Phi}\left(z\right)-\frac{\varphi \left(z\right)}{z-\frac{1}{2}}\right)}\text{}{e}^{-{\rho}_{z}\left(x\right)}$ | (18) | [49,50] |

_{0}, its median, σ its standard deviation, σ² its variance, a, b, β and s scale parameters, φ the normalized normal distribution, ϕ the normal law, Φ the cumulative normal law, z the degree of robustness and ρ

_{z}the Huber loss. The Huber loss depends on the degree of robustness and can be written following Equation (19) [49].

**Table 4.**Split subsets; T is for training, V for validation, S for stop, T

_{e}for test; ${C}_{{P}_{q}}$ is for the score calculated on the q subset.

1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | Scores |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|

V | T | T | T | T | T | T | T | T | T | T | S | T | T | T_{e} | ${C}_{{P}_{1}}$ |

T | V | T | T | T | T | T | T | T | T | T | S | T | T | T_{e} | ${C}_{{P}_{2}}$ |

T | T | V | T | T | T | T | T | T | T | T | S | T | T | T_{e} | ${C}_{{P}_{3}}$ |

T | T | T | V | T | T | T | T | T | T | T | S | T | T | T_{e} | ${C}_{{P}_{4}}$ |

… | |||||||||||||||

T | T | T | T | T | T | T | T | T | T | V | S | T | T | T_{e} | ${C}_{{P}_{11}}$ |

T | T | T | T | T | T | T | T | T | T | T | S | V | T | T_{e} | ${C}_{{P}_{13}}$ |

T | T | T | T | T | T | T | T | T | T | T | S | T | V | T_{e} | ${C}_{{P}_{14}}$ |

Median |

Model Element | Selected Hyperparameters | Tested Range Values | ||
---|---|---|---|---|

Order | r (L_{CV}) | 3 | (3–6) | (8–14) |

Exogenous input window-widths | n_{1} (I) | 8 | (7–10) | |

n_{2} (PET) | 12 | (9–12) | (9–12) | |

n_{3} (D_{SM}) | 5 | (2–5) | ||

n_{4} (D_{BP}) | 5 | (2–5) | (2–5) | |

n_{5} (R_{GC}) | 2 | (1–4) | (7–12) | |

n_{6} (R_{TB}) | 2 | (1–4) | (7–12) | |

n_{7} (R_{MA}) | 3 | (1–4) | (7–12) | |

Number of hidden neurons | N | 3 | (2–10) | (2–10) |

**Table 6.**Pearson’s correlation coefficients between the distributions with negative slope of groundwater evolution, and the theoretical composite pdf; ${r}_{in}^{2}$ applies to measurements inside prediction interval, ${r}_{out}^{2}$ applies to measurements outside the prediction interval.

Law | Normal | Gumbel | Laplace | Raised Cosine | Cauchy | Logistic | Slash | Bhatta-Charjee | Huber |
---|---|---|---|---|---|---|---|---|---|

${r}_{in}^{2}$ | 0.62 | 0.74 | 0.66 | 0.76 | 0.69 | 0.75 | 0.64 | 0.64 | 0.50 |

${r}_{out}^{2}$ | 0.68 | 0.74 | 0.67 | 0.77 | 0.70 | 0.75 | 0.72 | 0.75 | 0.65 |

**Table 7.**Pearson’s correlation coefficients between the distributions with positive slope of groundwater evolution and the theoretical composite pdf; ${r}_{in}^{2}$ applies to measurements inside prediction interval, ${r}_{out}^{2}$ applies to measurements outside the prediction interval.

Law | Normal | Gumbel | Laplace | Raised Cosine | Cauchy | Logistic | Slash | Bhatta-Charjee | Huber |
---|---|---|---|---|---|---|---|---|---|

${r}_{in}^{2}$ | 0.43 | 0.41 | 0.47 | 0.44 | 0.45 | 0.44 | 0.43 | 0.43 | 0.42 |

${r}_{out}^{2}$ | 0.52 | 0.54 | 0.54 | 0.66 | 0.55 | 0.61 | 0.63 | 0.57 | 0.52 |

**Table 8.**Error margin (${E}_{M}$ ), and Pearson’s correlation coefficients (${r}_{C}^{2})$ between the model of correctness and the empirical C

_{PICP}calculated for each 20-cm groundwater levels having a positive slope.

Law | Normal | Gumbel | Laplace | Raised Cosine | Cauchy | Logistic | Slash | Bhatta-charjee | Huber |
---|---|---|---|---|---|---|---|---|---|

${r}_{C}^{2}$ | 0.15 | 0.04 | 0.24 | 0.19 | 0.25 | 0.24 | 0.22 | 0.16 | 0.20 |

${E}_{M}$ | 76.3% | 73.7% | 72.4% | 76.3% | 75.0% | 73.7% | 80.3% | 76.3% | 77.6% |

**Table 9.**Error margin (${M}_{E}$ ), and Pearson’s correlation coefficients (${r}_{C}^{2})$ between the model of correctness and the empirical C

_{PICP}calculated for each class of 20-cm groundwater levels having a negative slope.

Law | Normal | Gumbel | Laplace | Raised Cosine | Cauchy | Logistic | Slash | Bhatta-charjee | Huber |
---|---|---|---|---|---|---|---|---|---|

${r}_{C}^{2}$ | 0.02 | 0.21 | 0.16 | 0.21 | 0.23 | 0.23 | 0.30 | 0.28 | 0.27 |

${M}_{E}$ | 74.4% | 70.3% | 70.3% | 72.5% | 70.3% | 79.1% | 71.4% | 73.6% | 70.3% |

**Table 10.**Pearson’s coefficients of correlation (r

^{2}) between the measured water level distribution and the composite pdf; correlations between the model correctness and the empirical C

_{PICP}calculated for each class of 20-cm groundwater levels (${r}_{C}^{2}$ ); and Error Margins (EM).

Groundwater Level Class | Criteria | Laws | |||
---|---|---|---|---|---|

Gumbel | Raised Cosine | Logistic | Slash | ||

Positive Slope | ${r}_{in}^{2}$ | 0.47 | 0.59 | 0.55 | 0.55 |

${r}_{out}^{2}$ | 0.52 | 0.56 | 0.53 | 0.54 | |

${r}_{C}^{2}$ | 0.19 | 0.33 | 0.20 | 0.15 | |

$\%EM$ | 60.5% | 61.8% | 60.5% | 63.2% | |

Negative Slope | ${r}_{in}^{2}$ | 0.78 | 0.79 | 0.77 | 0.72 |

${r}_{out}^{2}$ | 0.74 | 0.79 | 0.77 | 0.71 | |

${r}_{C}^{2}$ | 0.52 | 0.41 | 0.49 | 0.47 | |

$\%EM$ | 70.3% | 72.5% | 71.4% | 67.0% |

Confidence Index | C_{PICP}(Train + Test Datasets) | C_{PICP}(Test Set) | C_{MPI} (m) | C_{MPI} (m)(Without Extreme Values) |
---|---|---|---|---|

0.60 | 0.60 | 0.42 | 2.51 | 2.20 |

0.65 | 0.65 | 0.45 | 2.80 | 2.45 |

0.70 | 0.70 | 0.52 | 3.18 | 2.78 |

0.75 | 0.75 | 0.58 | 3.75 | 3.28 |

0.80 | 0.81 | 0.62 | 4.66 | 4.08 |

0.85 | 0.86 | 0.68 | 6.14 | 5.37 |

0.90 | 0.91 | 0.81 | 8.47 | 7.42 |

0.95 | 0.95 | 0.94 | 17.44 | 15.27 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Akil, N.; Artigue, G.; Savary, M.; Johannet, A.; Vinches, M. Uncertainty Estimation in Hydrogeological Forecasting with Neural Networks: Impact of Spatial Distribution of Rainfalls and Random Initialization of the Model. *Water* **2021**, *13*, 1690.
https://doi.org/10.3390/w13121690

**AMA Style**

Akil N, Artigue G, Savary M, Johannet A, Vinches M. Uncertainty Estimation in Hydrogeological Forecasting with Neural Networks: Impact of Spatial Distribution of Rainfalls and Random Initialization of the Model. *Water*. 2021; 13(12):1690.
https://doi.org/10.3390/w13121690

**Chicago/Turabian Style**

Akil, Nicolas, Guillaume Artigue, Michaël Savary, Anne Johannet, and Marc Vinches. 2021. "Uncertainty Estimation in Hydrogeological Forecasting with Neural Networks: Impact of Spatial Distribution of Rainfalls and Random Initialization of the Model" *Water* 13, no. 12: 1690.
https://doi.org/10.3390/w13121690