# Forecasting Multiple Groundwater Time Series with Local and Global Deep Learning Networks

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Data and Study Area

## 3. Methods

#### 3.1. Overview of Our Approach

- (1)
- the time series are first clustered and then a recurrent neural network model specifically designed for the time series (the LSTM, or long short-term memory model, [14]) is applied to each cluster,
- (2)
- a global, standard neural network (MLP) is created using all the time series, and
- (3)
- a global, recurrent neural network (LSTM-based) is created using all the time series with the DeepAR algorithm [15]. As discussed in more detail in Section 3.3.4, DeepAR is a global time series prediction algorithm that combines elements of LSTM modelling with classical time series autocorrelation structures, with the added capability of producing probabilistic predictions.

#### 3.2. Single Well Analyses

#### 3.2.1. Classical Statistical Formulation

_{t}, we have a set of predictors (or features). denoted X

_{t}, that can be used in constructing a prediction model. For our project, we consider dynamic regression models which take the following form:

_{1}, Y

_{2}, …, Y

_{T}} where Y

_{t}represents the well water level measured at time t, B is a vector of regression coefficients to be estimated and e

_{t}is an error term that can incorporate additional autocorrelation structure, if needed. In practice, the general philosophy is that the autocorrelation observable in a set of time series data can often be explained by measuring the right features. The best and most reliable predictions, particularly for the longer term, will result from models that incorporate a rich and relevant set of additional features. In [10], results obtained through this type of classic statistical time series modelling were compared with those obtained through the application of machine learning strategies.

#### 3.2.2. Multi-Layer Perceptron (MLP)

_{t}used to predict the outcome Y

_{t}at time t can also include information on X and Y values from times (t − 1), (t − 2) … and so on. Of course, this strategy will not work easily in settings where the time series has been measured sporadically.

#### 3.3. Multi-Well Analyses

#### 3.3.1. Self-Organising Map (SOM)

#### 3.3.2. Long Short-Term Memory Algorithm (LSTM)

^{th}memory cell) are modelled as:

#### 3.3.3. Global MLP Model

#### 3.3.4. Global Time Series Model (DeepAR)

#### 3.4. Summary of Methods

#### 3.5. Software

## 4. Modelling Setup and Results

#### 4.1. Single Well Analyses

#### 4.2. Multi-Well Analyses

#### 4.2.1. Partitioned Analysis with SOM and LSTMs

#### Self-Organising Maps

#### LSTMs

#### 4.2.2. Global MLP

#### 4.2.3. DeepAR (Global LSTM)

#### 4.3. Summary of Results

## 5. Discussion

## 6. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## Appendix A

## References

- Hiscock, K.M.; Bense, V.F. Hydrogeology: Principles and Practice; John Wiley & Sons: Hoboken, NJ, USA, 2021. [Google Scholar]
- Shen, C. A transdisciplinary review of deep learning research and its relevance for water resources scientists. Water Resour. Res.
**2018**, 54, 8558–8593. [Google Scholar] [CrossRef] - Shen, C.; Laloy, E.; Elshorbagy, A.; Albert, A.; Bales, J.; Chang, F.-J.; Ganguly, S.; Hsu, K.-L.; Kifer, D.; Fang, Z.; et al. HESS Opinions: Incubating deep-learning-powered hydrologic science advances as a community. Hydrol. Earth Syst. Sci.
**2018**, 22, 5639–5656. [Google Scholar] [CrossRef] [Green Version] - Zounemat-Kermani, M.; Mahdavi-Meymand, A.; Hinkelmann, R. A comprehensive survey on conventional and modern neural networks: Application to river flow forecasting. Earth Sci. Inform.
**2021**, 14, 893–911. [Google Scholar] [CrossRef] - Shen, C.; Chen, X.; Laloy, E. Broadening the use of machine learning in hydrology. Front. Water
**2021**, 3, 38. [Google Scholar] [CrossRef] - Reichstein, M.; Camps-Valls, G.; Stevens, B.; Jung, M.; Denzler, J.; Carvalhais, N.; Prabhat. Deep learning and process understanding for data-driven Earth system science. Nature
**2019**, 566, 195–204. [Google Scholar] [CrossRef] - Lee, S.; Lee, K.-K.; Yoon, H. Using artificial neural network models for groundwater level forecasting and assessment of the relative impacts of influencing factors. Hydrogeol. J.
**2019**, 27, 567–579. [Google Scholar] [CrossRef] - Nearing, G.S.; Kratzert, F.; Sampson, A.K.; Pelissier, C.S.; Klotz, D.; Frame, J.M.; Prieto, C.; Gupta, H.V. What role does hydrological science play in the age of machine learning? Water Resour. Res.
**2021**, 57, e2020WR028091. [Google Scholar] [CrossRef] - Gaffoor, Z.; Pietersen, K.; Jovanovic, N.; Bagula, A.; Kanyerere, T. Big data analytics and its role to support groundwater management in the southern African development community. Water
**2020**, 12, 2796. [Google Scholar] [CrossRef] - Clark, S.; Hyndman, R.J.; Pagendam, D.; Ryan, L.M. Modern strategies for time series regression. Int. Stat. Rev.
**2020**, 88, S179–S204. [Google Scholar] [CrossRef] - Montero-Manso, P.; Hyndman, R.J. Principles and algorithms for forecasting groups of time series: Locality and globality. arXiv
**2020**, arXiv:2008.00444. [Google Scholar] [CrossRef] - Bandara, K.; Bergmeir, C.; Smyl, S. Forecasting across time series databases using recurrent neural networks on groups of similar series: A clustering approach. Expert Syst. Appl.
**2019**, 140, 112896. [Google Scholar] [CrossRef] [Green Version] - Hewamalage, H.; Bergmeir, C.; Bandara, K. Recurrent neural networks for time series forecasting: Current status and future directions. Int. J. Forecast.
**2020**, 37, 388–427. [Google Scholar] [CrossRef] - Hochreiter, S.; Schmidhuber, J. LSTM can solve hard long time lag problems. Adv. Neural Inf. Process. Syst.
**1997**, 9, 473–479. [Google Scholar] - Salinas, D.; Flunkert, V.; Gasthaus, J.; Januschowski, T. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. Int. J. Forecast.
**2019**, 36, 1181–1191. [Google Scholar] [CrossRef] - Kratzert, F.; Klotz, D.; Shalev, G.; Klambauer, G.; Hochreiter, S.; Nearing, G. Towards learning universal, regional, and local hydrological behaviors via machine learning applied to large-sample datasets. Hydrol. Earth Syst. Sci.
**2019**, 23, 5089–5110. [Google Scholar] [CrossRef] [Green Version] - Yin, W.; Fan, Z.; Tangdamrongsub, N.; Hu, L.; Zhang, M. Comparison of physical and data-driven models to forecast groundwater level changes with the inclusion of GRACE—A case study over the state of Victoria, Australia. J. Hydrol.
**2021**, 602, 126735. [Google Scholar] [CrossRef] - Clark, S.R. Unravelling groundwater time series patterns: Visual analytics-aided deep learning in the Namoi region of Australia. Environ. Model. Softw.
**2022**, 149, 105295. [Google Scholar] [CrossRef] - WaterNSW. Available online: https://realtimedata.waternsw.com.au (accessed on 1 March 2020).
- SILO—Australian Climate Data from 1889 to Yesterday. Available online: https://www.longpaddock.qld.gov.au/silo/ (accessed on 1 March 2020).
- Cressie, N.; Wikle, C.K. Statistics for Spatio-Temporal Data; John Wiley & Sons: Hoboken, NJ, USA, 2015. [Google Scholar]
- Kohonen, T. The self-organizing map. Proc. IEEE
**1990**, 78, 1464–1480. [Google Scholar] [CrossRef] - Kohonen, T. Essentials of the self-organizing map. Neural Netw.
**2013**, 37, 52–65. [Google Scholar] [CrossRef] - Clark, S.; Sisson, S.A.; Sharma, A. Tools for enhancing the application of self-organizing maps in water resources research and engineering. Adv. Water Resour.
**2020**, 143, 103676. [Google Scholar] [CrossRef] - Elman, J.L. Finding structure in time. Cogn. Sci.
**1990**, 14, 179–211. [Google Scholar] [CrossRef] - Goodfellow, I.; Bengio, Y.; Courville, A. Deep Learning; MIT Press: Cambridge, MA, USA, 2016. [Google Scholar]
- Vlachas, P.R.; Byeon, W.; Wan, Z.Y.; Sapsis, T.P.; Koumoutsakos, P. Data-driven forecasting of high-dimensional chaotic systems with long short-term memory networks. Proc. R. Soc. A Math. Phys. Eng. Sci.
**2018**, 474, 20170844. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Chollet, F.; Allaire, J.; Falbel, D.; Tang, Y.; Van Der Bijl, W.; Studer, M. R Interface to Keras; Keras Team. 2017. Available online: https://cloud.r-project.org/web/packages/keras/index.html (accessed on 1 April 2020).
- R Core Team. R: A Language and Environment for Statistical Computing; R Foundation for Statistical Computing: Vienna, Austria, 2020; Available online: https://www.R-project.org/ (accessed on 1 April 2020).
- Wehrens, R.; Buydens, L.M. Self-and super-organizing maps in R: The Kohonen package. J. Stat. Softw.
**2007**, 21, 1–19. [Google Scholar] [CrossRef] [Green Version] - GluonTS—Probabilistic Time Series Modeling. Available online: https://ts.gluon.ai/ (accessed on 8 June 2021).
- DeepAR Forecasting Algorithm. Available online: https://docs.aws.amazon.com/sagemaker/latest/dg/deepar.html (accessed on 8 June 2021).
- Makridakis, S.; Spiliotis, E.; Assimakopoulos, V. M5 accuracy competition: Results, findings, and conclusions. Int. J. Forecast.
**2022**, in press. [Google Scholar] [CrossRef] - Kratzert, F.; Klotz, D.; Brenner, C.; Schulz, K.; Herrnegger, M. Rainfall–runoff modelling using Long Short-Term Memory (LSTM) networks. Hydrol. Earth Syst. Sci.
**2018**, 22, 6005–6022. [Google Scholar] [CrossRef] [Green Version] - Willard, J.D.; Read, J.S.; Appling, A.P.; Oliver, S.K.; Jia, X.; Kumar, V. Predicting water temperature dynamics of unmonitored lakes with meta-transfer learning. Water Resour. Res.
**2021**, 57, e2021WR029579. [Google Scholar] [CrossRef] - GRACE Tellus—Gravity Recovery & Climate Experiment, NASA. Available online: https://grace.jpl.nasa.gov/ (accessed on 15 February 2022).
- Rudin, C.; Radin, J. Why Are We Using Black Box Models in AI When We Don’t Need to? A Lesson from an Explainable AI Competition. 2019. Available online: https://hdsr.mitpress.mit.edu/pub/f9kuryi8/release/7 (accessed on 15 January 2022).
- Durbin, J.; Koopman, S.J. Time Series Analysis by State Space Methods; OUP Oxford: Oxford, UK, 2012. [Google Scholar]

**Figure 1.**Observed (dark blue) and predicted (orange) groundwater levels from a classical time series model (upper plot) and a neural network model (lower plot). The classical prediction includes 80 and 95% prediction intervals, also shaded in orange. Adapted with permission from [10]. Copyright 2020 John Wiley & Sons.

**Figure 2.**Monitoring sites and environmental monitoring stations used in this study. Adapted with permission from ref [18]. Copyright 2022 Elsevier.

**Figure 3.**A sample of the 165 groundwater time series that make up the data set. Varying temporal patterns and record lengths are evident among the time series.

**Figure 4.**Measurement frequency at the 165 groundwater bores in the data set. Bores are represented by columns arranged from left to right by site number, which roughly follows date of station commissioning. For each bore, the months of station operation are represented vertically. Yellow indicates months with >25 measurements (i.e., daily), identifying the period of telemetry for some of the bores. White indicates no measurements.

**Figure 5.**Groundwater hydrograph and some associated predictors for a single well (GW030344_2). Non-parametric smoothers have been added (red lines) to make any long-term trends more evident.

**Figure 7.**Individual well MLP daily predictions (blue, dashed) and observations (black). RMSE = 0.134.

**Figure 8.**MLP predictions for two example wells (blue) over observed data (black), and predicted water level if extraction data are set to zero (orange).

**Figure 9.**Comparison of MLP predictions (blue, dashed) over observed data (black) using daily data (upper panel) and monthly data (lower panel) for an individual well.

**Figure 10.**SOM clusters, identifying groups of wells with similar temporal patterns. The upper panel shows the 16 most prevalent groundwater level patterns found in the dataset. Coloured smoothers are added to indicate the predominant trend in each cluster, with similar colours indicating more similar patterns. The lower portion shows examples of measurements taken at wells in three of these clusters, with data points coloured by measurement well (there are 9, 11, and 10 wells allocated to clusters 1, 12 and 13 respectively). The colours in the lower panels are present to identify the measurements taken at one well from those at another and do not relate to the colours in the upper panel. Adapted with permission from ref. [18]. Copyright 2022 Elsevier.

**Figure 11.**Visual analysis of historical data with the SOM. Well locations are coloured by the SOM cluster that is best matched by their historical time series (1974–2018). The time series’ general patterns for each colour are shown in the legend. Adapted with permission from [18]. Copyright 2022 Elsevier.

**Figure 12.**LSTM prediction (red) for one cluster shown with measurement data from an un-telemetered well in the Upper Namoi region (upper panel, blue dots), and an untelemetered well in the Lower Namoi region (lower panel, green dots), showing that the same LSTM prediction can apply to wells in different regions that are members of the same cluster. Adapted with permission from [18]. Copyright 2022 Elsevier.

**Figure 13.**Global MLP groundwater level predictions for 165 time series (grouped and coloured here by SOM cluster as described in Figure 10) shown over observations (black). RMSE = 0.239.

**Figure 14.**Comparison of monthly predictions from the global MLP (upper panel, RMSE = 0.114) and individual monthly MLP (lower panel, RMSE = 0.156) for the same well. Predictions are in dashed blue and observations in black.

**Figure 15.**Prediction on an untelemetered well using global MLP (blue, dashed) over measured data (black). RMSE = 0.195.

**Figure 16.**DeepAR predictions for the last 5 years of each of the 165 time series arranged and coloured by SOM cluster membership (as described in Figure 10) with observed data in grey. Top row: entire study duration (1974–2018); Lower row: final 5 years of study (January 2014–December 2018). RMSE = 0.209.

**Figure 17.**DeepAR predictions for the last 10 years of the study period, arranged and coloured by SOM cluster membership (as described in Figure 10) with observed data in grey (RMSE = 0.477).

**Figure 18.**Top panel: Observed data (blue dots) for first 200 datapoints, along with fitted values from the linear regression models. Middle panel shows observed data and predicted values from the two linear models, as well as the ARIMA model for the short-term prediction scenario (days 201–400). Bottom panel shows the observed data and predicted values from the two linear models as well as the ARIMA model for the long-term prediction scenario (days 1800–2000). In all panels, OLS predictions are shown in black, linear model with lags in green, and ARIMA in red.

**Table 1.**Summary of algorithms and input data used in the study. Frequencies of observations used in the models are denoted by D = daily, M = monthly or A = annual frequency.

Algorithm | Purpose |
Response Data (Groundwater Levels) | Predictors |
---|---|---|---|

MLP (individual) | Input/output predictions | Individual time series | Rainfall (+30D, 12M lags) (D, M), Evapotranspiration (D, M), Surface flows (+30D, 12M lags) (D, M), Extractions (A), Day-of-study, Month-of-year |

SOM | Unsupervised clustering and dimension reduction | Clusters of time series | All of the groundwater time series (M) |

LSTM | Deep learning time series predictions | Representative time series from each cluster | Rainfall (M), Evapotranspiration (M), Surface flows (M), Extractions (A), Month-of-year, (Lookback varies by cluster) |

Global MLP | Input/output predictions for multiple monitoring data sets | All time series | Rainfall (+12M lags) (M), Evapotranspiration (M), Surface flows (+12M lags) (M), Extractions (A), Well ID, Month-of-study, Month-of-year |

DeepAR | Multiple time series probabilistic predictions | All time series | Rainfall (+12M lags) (M), Evapotranspiration (M), Surface flows (M), Extractions (A), Well ID, Year-of-study (60-month context length) |

**Table 2.**Summary of results in terms of root mean square error (RMSE) for the various modelling strategies employed.

Method | Aggregation | Number of Series |
Number of Data Points |
Lagged Predictors | Average RMSE ** |
---|---|---|---|---|---|

Individual MLP | Monthly | 11 * | 299 (mean per series) | 12 months rain lags | 0.195 |

12 months rain and surface flow lags | 0.165 | ||||

Daily | 11 * | 3212 (mean per series) | 30 days rain lags | 0.163 | |

30 days rain and surface flow lags | 0.153 | ||||

Global MLP | Monthly | 165 | 36,142 | 12 months rain and surface flow lags | 0.239 |

Partitioned LSTMs | Monthly | 16 ^ | 540 (per cluster) | Various lookback lengths | 0.198 |

DeepAR (5 year) | Monthly | 165 | 24,839 (47 predictors, 528 months) | 12 months rain lags | 0.209 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Clark, S.R.; Pagendam, D.; Ryan, L.
Forecasting Multiple Groundwater Time Series with Local and Global Deep Learning Networks. *Int. J. Environ. Res. Public Health* **2022**, *19*, 5091.
https://doi.org/10.3390/ijerph19095091

**AMA Style**

Clark SR, Pagendam D, Ryan L.
Forecasting Multiple Groundwater Time Series with Local and Global Deep Learning Networks. *International Journal of Environmental Research and Public Health*. 2022; 19(9):5091.
https://doi.org/10.3390/ijerph19095091

**Chicago/Turabian Style**

Clark, Stephanie R., Dan Pagendam, and Louise Ryan.
2022. "Forecasting Multiple Groundwater Time Series with Local and Global Deep Learning Networks" *International Journal of Environmental Research and Public Health* 19, no. 9: 5091.
https://doi.org/10.3390/ijerph19095091